PIPE IO Technical Bulletin

In this document the lower case “pipe” refers to a UNIX pipe and the upper case “PIPE” to the GT.M device.

The V5.3-003 release extends the capabilities of the I/O status variables with support for a new PIPE device on the UNIX platforms as a mnemonic space device. An OPEN of the device starts a sub-process. Data written to the device by the M program is available to the process on its STDIN. The M program can read the STDOUT and STDERR of the sub-process. This facilitates output only applications, such as printing directly from a GT.M program to an lp command; input only applications, such as reading the output of a command such as ps; and co-processing applications, such as using iconv to convert data from one encoding to another. (C9H05-002859)

GT.M provides several I/O status variables that convey information about the status of individual operations.

A PIPE is akin to a FIFO device. Both FIFO and PIPE map GT.M devices to UNIX pipes, the conceptual difference being that whereas a FIFO device specifies a named pipe, but does not specify the process on the other end of the pipe, a PIPE device specifies a process to communicate with, but the pipes are unnamed. Specifically, an OPEN of a PIPE creates a subprocess with which the GT.M process communicates.

A PIPE device is specified with a "PIPE" value for mnemonicspace on an OPEN command.

	GT.M ignores the mnemonicspace specification on an OPEN of a previously OPEN device and leaves the existing device with its original characteristics.

The OPEN command for a PIPE provides a number of variations in the use of UNIX pipes shown below as Examples 1-4.

Example 1

       
            SET p="Printer"
            OPEN p:(COMMAND="lp":WRITEONLY)::"PIPE" 
        

Example 1 shows the use of a PIPE device to spool data to the default printer by spooling to the lp command, opened via the default shell (the shell specified by the SHELL environment variable, and the shell used to start GT.M if SHELL is unspecified). The WRITEONLY device parameter specifies that the GT.M process will not read data back from the lpr command.

Example 2

            Set p="MyProcs"
            OPEN p:(COMMAND="ps -ef|grep $USER":READONLY)::"PIPE" 
        

Example 2 shows the use of a PIPE device to identify processes belonging to the current userid. The READONLY device parameter specifies that the GT.M process will only read the output of the pipe, and will not provide it with any input. This example illustrates the fact that the command can be any shell command, can include environment variables and pipes within the command.

	On your UNIX, flags to the ps command may differ from this example.

Example 3:

            Set p="Convert"
            OPEN p:(SHELL="/bin/csh":COMMAND="iconv -f ISO_8859-1 -t WINDOWS-1252")::"PIPE"  
        

Example 3 shows the use of a process to whose input the GT.M process writes to and whose output the GT.M process reads back in, in this example convering data from an ISO 8859-1 encoding to the Windows 1252 encoding. This example also shows the use of a different shell from the default. If the OPEN deviceparameters don't specify a SHELL, the PIPE device uses the shell specified by the environment variable SHELL; if it does not find a definition for SHELL, the device uses the system default /bin/sh.

Example 4:

            SET p="Files"
            SET e="Errors"
            OPEN p:(COMMAND="find /var/log -type d -print":READONLY:STDERR=e)::"PIPE"  
        

Example 4, GT.M uses the standard system utility find to obtain a list of subdirectories of /var/log, which are read back via the device with handle "Files" with any errors (e.g, “Permission denied” messages for sub-directories that the find command cannot process) read back via the device with handle "Errors".

The PIPE device does non-blocking writes. If a process tries to WRITE to a full PIPE and the WRITE would block the device implicitly tries to complete the operation up to10 times. If the implicit retries don't succeed (remain blocked), the WRITE sets $DEVICE to "1,Resource temporarily unavailable", $ZA to 9, and triggers an error. If the GT.M process has defined an EXCEPTION, $ETRAP or $ZTRAP, it may choose to retry the WRITE after some action or delay might remove data from the pipe.

The CLOSE of a PIPE device prevents all subsequent access to the pipes associated with the device. Unless the OPEN that created the device specified INDEPENDENT, the process terminates. Note that any subsequent attempt by the created process to read from its stdin (which would be a closed pipe) returns an EOF and typical UNIX behavior would be to terminate on such an event.

The following examples show the use of deviceparameters and status variables with PIPE devices.

Example :

            SET p1="test1" 
            OPEN p1:(shell="/bin/sh":comm="cat")::"PIPE"
            FOR i=1:1:10 DO
            . USE p1 
            . WRITE i,":abcdefghijklmnopqrstuvwxyz abcdefghijklmnopqrstuvwxyz ",! 
            . READ x
            . USE $P 
            . WRITE x,! 
            CLOSE p1
            QUIT         
        

This WRITEs 10 lines of output to the cat command and reads the cat output back into the local variable x. The GT.M process WRITEs each line READ from the PIPE to the principal device. This example works since “cat” is not a buffering command. The example above would not work for a command such as tr that buffers its input.

Example :

            SET p1="test1"
            OPEN  p1:(SHELL="/bin/sh":COMMAND="tr -d e")::"PIPE"
            FOR i=1:1:1000 DO
            . USE p1
            . WRITE i,":abcdefghijklmnopqrstuvwxyz abcdefghijklmnopqrstuvwxyz ",!
            . READ x:0
            . IF '+$DEVICE USE $P WRITE x,! 
            USE p1
            WRITE /EOF
            FOR  READ x QUIT:$ZEOF  USE $P WRITE x,! USE p1
            CLOSE p1
            QUIT   
        

This shows the use of “tr” (a buffering command) in the created process for the PIPE device. To see the buffering effect the GT.M process WRITEs 1000 lines to the PIPE device. Different operating systems may have different buffer sizes. Notice the use of the r x:0 and the check on $DEVICE in the loop. If $DEVICE is 0, WRITE x writes the data read to the principal device. No actual READs complete, however, until “tr” reaches its buffer size and writes to its stdout. The final few lines remain buffered by “tr” after the process finishes the first loop. The GT.M process then issues a WRITE /EOF to the PIPE causing “tr” to flush its buffered lines. In the final for loop the GT.M process uses the simple form of READ x from the PIPE followed by a WRITE of each line to the principal device until $zeof becomes TRUE.

Example :

 
            SET a="test"
            OPEN a:(COMMMAND="ntestin":INDEPENDENT)::"PIPE"
            USE a
            SET key=$KEY
            WRITE "Show ntestin still running after CLOSE of a",!
            WRITE "The parent process of 1 shows the parent shell has exited after CLOSE of a"
            READ line1,line2
            USE $p
            WRITE !,line1,!,line2,!,!
            SET k="ps -ef | grep -v grep | grep -v sh | grep -w '"_key_"' | awk '{print $2}'"
            SET b="getpid"
            OPEN b:(COMMMAND=k:READONLY)::"PIPE"
            USE b
            READ pid
            CLOSE a
            CLOSE b
            SET k2="ps -ef | grep -v grep | grep -v sh | grep -w '"_pid_"'"
            SET c="psout"
            OPEN c:(COMMMAND=k2:WRITEONLY)::"PIPE"
            CLOSE c
            QUIT  
        

This demonstrates that the created process “ntestin” keeps running as an INDEPENDENT process after the GT.M process CLOSEs the pipe. This GT.M process uses another PIPE device to return the process id of “ntestin” and READ it into “pid” so that it may be killed by this or another process, should that be appropriate.

Example :

            SET p1="test1"
            SET a=0
            OPEN p1:(SHELL="/bin/sh":COMMAND="cat":EXCEPTION="GOTO cont1")::"PIPE"
            SET c=":abcdefghijklmnopqrstuvwxyz abcdefghijklmnopqrstuvwxyz"
            FOR i=1:1:10000  DO
            . USE p1
            . WRITE i_c,!
            . USE $p WRITE i,!
            USE p1
            WRITE /EOF
            FOR  READ x QUIT:$ZEOF  USE $p WRITE x,! USE p1
            CLOSE p1
            QUIT
            cont1
            IF $ZEOF QUIT
            IF a=0 SET a=i/2
            SET z=$ZA
            ; use $device to make sure ztrap is caused by blocked write to pipe 
            SET d=$DEVICE 
            IF "1,Resource temporarily unavailable"=d DO 
            . USE $p
            . WRITE "pipe full, i= ",i," $ZA = ",z,!
            . SET i=i-1 
            . USE p1
            . FOR j=1:1:a  READ x USE $p WRITE j,"-",x,! USE p1
            QUIT  
        

This demonstrates WRITEs to a PIPE device with blocking. The WRITE loop has no READ to force the input pipe to fill up which blocks the cat output, causing cat to stop reading its input, letting the pipe acting as input on the PIPE device to fill up and creating the blocked condition. When the process takes the $ZTRAP to cont1 it tests $DEVICE to determine if the trap is caused by the full pipe. If so, it uses the for loop to read half the number of lines output by the main loop. It decrements “i” and returns to the original WRITE loop to retry the failed line and continue with the WRITEs to the pipe. Depending upon the configuration of the environment, it may trap several times before processing all lines.