Using PIPE Devices

A PIPE device is used to access and manipulate the input and/or output of a shell command as a GT.M I/O device. GT.M maintains I/O status variables for a PIPE device just as it does for other devices. An OPEN of the device starts a sub-process. Data written to the device by the M program is available to the process on its STDIN. The M program can read the STDOUT and STDERR of the sub-process. This facilitates output only applications, such as printing directly from a GT.M program to an lp command; input only applications, such as reading the output of a command such as ps; and co-processing applications, such as using iconv to convert data from one encoding to another.

A PIPE is akin to a FIFO device. Both FIFO and PIPE map GT.M devices to UNIX pipes, the conceptual difference being that whereas a FIFO device specifies a named pipe, but does not specify the process on the other end of the pipe, a PIPE device specifies a process to communicate with, but the pipes are unnamed. Specifically, an OPEN of a PIPE creates a subprocess with which the GT.M process communicates.

A PIPE device is specified with a "PIPE" value for mnemonicspace on an OPEN command.

[Note]Note

GT.M ignores the mnemonicspace specification on an OPEN of a previously OPEN device and leaves the existing device with its original characteristics.

The OPEN command for a PIPE provides a number of variations in the use of UNIX pipes shown below as Examples 1-4.

Example:

set p="Printer"
open p:(command="lpr":writeonly)::"PIPE"

This shows the use of a PIPE device to spool data to the default printer by spooling to the lpr command, opened via the default shell (the shell specified by the SHELL environment variable, and the shell used to start GT.M if SHELL is unspecified). The WRITEONLY device parameter specifies that the GT.M process not read data back from the lpr command. Use WRITEONLY when no errors are expected from the application(s) in the pipe. WRITEONLY tends not to serve most applications well.

Example: 

set p="MyProcs"
open p:(command="ps -ef|grep $USER":readonly)::"PIPE"

This shows the use of a PIPE device to identify processes belonging to the current userid. The READONLY device parameter specifies that the GT.M process only read the output of the pipe, and not provide it with any input. This example illustrates the fact that the command can be any shell command, can include environment variables and pipes within the command.

[Note]Note

Flags to the ps command vary for different UNIX platforms.

Example:

set p="Convert"
open p:(shell="/bin/csh":command="iconv -f ISO_8859-1 -t WINDOWS-1252")::"PIPE"

This shows the use of a process to whose input the GT.M process writes to and whose output the GT.M process reads back in, in this example converting data from an ISO 8859-1 encoding to the Windows 1252 encoding. This example also shows the use of a different shell from the default. If the OPEN deviceparameters don't specify a SHELL, the PIPE device uses the shell specified by the environment variable SHELL; if it does not find a definition for SHELL, the device uses the system default /bin/sh.

Example:

set p="Files"
set e="Errors"
open p:(command="find /var/log -type d -print":readonly:stderr=e)::"PIPE"

GT.M uses the standard system utility find to obtain a list of subdirectories of /var/log, which are read back via the device with handle "Files" with any errors (for example, "Permission denied" messages for sub-directories that the find command cannot process) read back via the device with handle "Errors".

The following characteristics of PIPE may be helpful in using them effectively.

With Read:

A READ with no timeout reads whatever data is available to be read; if there is no data to be read, the process hangs until some data becomes available.

A READ with a timeout reads whatever data is available to be read, and returns; if there is no data to be read, the process waits for a maximum of the timeout period, an integer number of seconds, for data to become available (if the timeout is zero, it returns immediately, whether or not any data was read). If the READ returns before the timeout expires, it sets $TEST to TRUE(1); if the timeout expires, it sets $TEST to FALSE (0). When the READ command does not specify a timeout, it does not change $TEST. READ specifying a maximum length (for example, READ X#10 for ten characters) reads until either the PIPE has supplied the specified number of characters, or a terminating delimiter.

The following table shows the result and values of I/O status variables for various READ operations on a PIPE device.

Operation

Result

$DEVICE

$ZA

$TEST

X

$ZEOF

READ X:n

Normal Termination

0

0

1

Data Read

0

READ X:n

Timeout with no data read

0

0

0

empty string

0

READ X:n

Timeout with partial data read

0

0

0

Partial data

0

READ X:n

End of File

1,Device detected EOF

9

1

empty string

1

READ X:0

Normal Termination

0

0

1

Data Read

0

READ X:0

No data available

0

0

0

empty string

0

READ X:0

Timeout with partial data read

0

0

0

Partial data

0

READ X:0

End of File

1,Device detected EOF

9

1

empty string

1

READ X

Error

1,<error signature>

9

n/c

empty string

0

With WRITE:

The PIPE device does non-blocking writes. If a process tries to WRITE to a full PIPE and the WRITE would block, the device implicitly tries to complete the operation up to a default of 10 times. If the gtm_non_blocked_write_retries environment variable is defined, this overrides the default number of retries. If the retries do not succeed (remain blocked), the WRITE sets $DEVICE to "1,Resource temporarily unavailable", $ZA to 9, and produces an error. If the GT.M process has defined an EXCEPTION, $ETRAP or $ZTRAP, the error trap may choose to retry the WRITE after some action or delay that might remove data from the PIPE device.

With WRITE /EOF:

WRITE /EOF to a PIPE device flushes, sets $X to zero (0) and terminates output to the created process, but does not CLOSE the PIPE device. After a WRITE /EOF, any additional WRITE to the device discards the content, but READs continue to work as before. A WRITE /EOF signals the receiving process to expect no further input, which may cause it to flush any output it has buffered and terminate. You should explicitly CLOSE the PIPE device after finishing all READs. If you do not want WRITE /EOF to flush any pending output including padding in FIXED mode or a terminating EOL in NOFIXED mode, SET $X=0 prior to the WRITE /EOF.

To avoid an indefinite hang doing a READ from a created process that buffers its output to the input of the PIPE device, READ with timeout (typically 0).

With CLOSE:

The CLOSE of a PIPE device prevents all subsequent access to the pipes associated with the device. Unless the OPEN that created the device specified INDEPENDENT, the process terminates. Note that any subsequent attempt by the created process to read from its stdin (which would be a closed pipe) returns an EOF and typical UNIX behavior would be to terminate on such an event.

The following examples show the use of deviceparameters and status variables with PIPE devices.

Example:

pipe1;
  set p1="test1" 
  open p1:(shell="/bin/sh":comm="cat")::"PIPE"
  for i=1:1:10 do
  . use p1 
  . write i,":abcdefghijklmnopqrstuvwxyz abcdefghijklmnopqrstuvwxyz ",! 
  . read x
  . use $P 
  . write x,! 
  close p1
  quit

This WRITEs 10 lines of output to the cat command and reads the cat output back into the local variable x. The GT.M process WRITEs each line READ from the PIPE to the principal device. This example works because "cat" is not a buffering command. The example above would not work for a command such as tr that buffers its input.

Example :

pipe3;
  set p1="test1"
  open p1:(shell="/bin/sh":command="tr -d e")::"PIPE"
  for i=1:1:1000 do
  . use p1
  . write i,":abcdefghijklmnopqrstuvwxyz abcdefghijklmnopqrstuvwxyz ",!
  . read x:0
  . if '+$device use $principal write x,! 
  use p1
  write /EOF
  for  read x quit:$zeof  use $principal write x,! use p1
  close p1
  quit

This shows the use of tr (a buffering command) in the created process for the PIPE device. To see the buffering effect the GT.M process WRITEs 1000 lines to the PIPE device. Different operating systems may have different buffer sizes. Notice the use of the r x:0 and the check on $DEVICE in the loop. If $DEVICE is 0, WRITE x writes the data read to the principal device. No actual READs complete, however, until tr reaches its buffer size and writes to its stdout. The final few lines remain buffered by tr after the process finishes the first loop. The GT.M process then issues a WRITE /EOF to the PIPE causing tr to flush its buffered lines. In the final for loop the GT.M process uses the simple form of READ x from the PIPE followed by a WRITE of each line to the principal device until $zeof becomes TRUE.

Example : 

pipe4;
  set a="test"
  open a:(command="nestin":independent)::"PIPE"
  use a 
  set key=$KEY
  write "Show nestin still running after CLOSE of a",!
  write "The parent process of 1 shows the parent shell has exited after CLOSE of a"
  read line1,line2
  use $principal
  write !,line1,!,line2,!,!
  set k="ps -ef | grep -v grep | grep -v sh | grep -w '"_key_"' | awk '{print $2}'"
  set b="getpid"
  open b:(command=k:readonly)::"PIPE"
  use b
  read pid
  close a
  close b
  set k2="ps -ef | grep -v grep | grep -v sh | grep -w '"_pid_"'"
  set c="psout"
  open c:(command=k2:writeonly)::"PIPE"
  close c
  quit

This demonstrates that the created sub process nestin keeps running as an INDEPENDENT process after the GT.M process CLOSEs the pipe. This GT.M process uses another PIPE device to return the process id of nestin and READ it into pid so that it may be killed by this or another process, should that be appropriate.

[Note]Note

"nestin.c" is a program which reads from standard input and writes to standard output until it see and EOF. It then loops for 300 1sec sleeps doing nothing. The purpose of using independent is as a server process which continues until it receives some other signal for termination.

Example:

GTM>kill ^a
GTM>zprint ^indepserver
indepserver;
  read x
  write "received = ",x,!
  set ^quit=0
  for  do  quit:^quit
  . if $data(^a) write "^a = ",^a,!
  . Hang 5
GTM>set a="test"
GTM>open a:(command="mumps -run ^indepserver>indout":independent)::"pipe"
GTM>use a
GTM>write "instructions",!
GTM>close a
GTM>zsystem "cat indout"
received = instructions
GTM>set ^a=1
GTM>zsystem "cat indout"
received = instructions
^a = 1
^a = 1
^a = 1
GTM>s ^quit=1
GTM>zsystem "cat indout"
received = instructions
^a = 1
^a = 1
^a = 1
^a = 1
GTM>

This is a simple example using a mumps process as a server.

Example: 

pipe5;
  set p1="test1"
  set a=0
  open p1:(shell="/bin/sh":command="cat":exception="goto cont1")::"PIPE"
  set c=":abcdefghijklmnopqrstuvwxyz abcdefghijklmnopqrstuvwxyz"
  for i=1:1:10000  do
  . use p1
  . write i_c,!
  . use $principal write i,!
  use p1
  write /EOF
  for  read x quit:$zeof  use $principal write x,! use p1
  close p1
  quit
cont1
  if $zeof quit
  if a=0 set a=i/2
  set z=$za
  ; use $device to make sure ztrap is caused by blocked write to pipe 
  set d=$device
  if "1,Resource temporarily unavailable"=d DO 
  . use $p
  . write "pipe full, i= ",i," $ZA = ",z,!
  . set i=i-1 
  . use p1
  . for j=1:1:a  read x use $principal write j,"-",x,! use p1
  quit

This demonstrates how to deal with write blocking of a PIPE device. The loop doing the WRITE does not READ from the PIPE. Eventually causing the output of cat to block on its output and stop reading input from the pipe. When the process takes the $ZTRAP to cont1 it tests $DEVICE to determine if the trap is caused by the full pipe. If so, it uses the for loop to read half the number of lines output by the main loop. It decrements i and returns to the original WRITE loop to retry the failed line and continue with the WRITEs to the pipe. Depending upon the configuration of the environment, it may trap several times before processing all lines.

Example:

  ; Example program that starts another program in a pipe and traps the errors. The called
; programs intentionally induce errors
pipexample
  set $etrap="do readfrompipe(.pipe,.piperr) use $p zwrite $zstatus zhalt 99"
  set pipe="pipe"
  set piperr="piperr"
  set writesize=1024
  set cmd=$piece($zcmdline," ") set:'$length(cmd) cmd="induceEPIPE"
  open pipe:(shell="/bin/bash":command="$gtm_dist/mumps -run "_cmd_"^pipexample":stderr=piperr)::"pipe"
  zshow "D":devicelist write "The active device is ",devicelist("D",2),!
  use pipe
  for i=1:1:1024 write $tr($justify(i,writesize)," ","X"),!
  close pipe
  quit
; Same as above, but without defining the PIPE's standard error
nostderr
  set $etrap="do readfrompipe(.pipe) use $p zshow ""*"" zhalt 99"
  set pipe="pipe"
  set writesize=1024
  set cmd=$piece($zcmdline," ",2) set:'$length(cmd) cmd="induceEAGAIN"
  open pipe:(shell="/bin/bash":command="$gtm_dist/mumps -run "_cmd_"^pipexample")::"pipe"
  zshow "D":devicelist write "The active device is ",devicelist("D",2),!
  write !,!
  use pipe
  for i=1:1:1024 write $tr($justify(i,writesize)," ","X"),!
  close pipe
  quit
; This routine intentionally delays reading from the pipe to induce an EAGAIN
induceEAGAIN
  set $etrap="use $p zwrite $zstatus zhalt 99"
  set hangtime=+$zcmdline set:'hangtime hangtime=5 set add=1
  for i=1:1:1024 read x(i) quit:$zeof  do
  . set delay(i)=1/(add+$random(hangtime))
  . hang delay(i)
  . set:i=30 add=10
  halt
; This routine intentionally induces an EPIPE by immediately sending a SIGTERM to itself causing
; a FORCEDHALT error which goes to STDERR. Subsequently, a random DIVZERO error occurs, but this
; error goes to STDOUT since it is not a fatal error
induceEPIPE
  set $etrap="use $p zwrite $zstatus zhalt 99"
  set divzero=150373210 ; DIVZERO goes to stdout
  write "My PID is ",$job,!
  zsystem:'$zcmdline "kill -15 "_$job  ; FORCEDHALT error goes to stderr
  for i=1:1 read x(i) quit:$zeof  zmessage:'$random(1000) divzero
  halt
; Read the contents of the pipe on failure. Messages from the programs inside the pipe aid
; in undestanding the underlying problem(s)
readfrompipe(pipe,piperr)
  new i
  new $etrap
  set $etrap="set x=$zjobexam() zhalt 88"
  use pipe
  for i=1:1  read pipe(i):0 quit:'$test!$zeof
  zkill pipe(i)
  do:$data(piperr)
  . use piperr
  . for i=1:1  read piperr(i):0 quit:'$test!$zeof
  . zkill piperr(i)
  close pipe
  use $p
  for i=1:1  quit:'$data(pipe(i))  write ?4,"stdout:",pipe(i),!
  for i=1:1  quit:'$data(piperr(i))  write ?4,"stderr:",piperr(i),!
  quit
; Example of trapping an error and retrying the operation as necessary. Error conditions
; used are EPIPE, aka "Broken pipe" or ENO32, and EAGAIN, aka ENO11.
retry
  set $etrap="use $p zshow ""*"" zhalt 99"
  set pipe="pipe"
  set piperr="piperr"
  set writesize=1024
  set cmd=$piece($zcmdline," ") set:'$length(cmd) cmd="induceEPIPE"
  for try=0:1  do  quit:$get(readcomplete,0)
  . new $etrap set $etrap="goto retryEPIPE"
  . open pipe:(shell="/bin/bash":command="$gtm_dist/mumps -run "_cmd_"^pipexample "_try:stderr=piperr)::"pipe"
  . zshow "D":devicelist write "Try ",try,$char(9),devicelist("D",2),!
  . use pipe
  . for i=1:1:1024 do
  . . new $etrap set $etrap="goto retryEAGAIN^pipexample"
  . . write $tr($justify(i,writesize)," ","X"),!
  . set readcomplete=1
  close pipe
  use $p
  write ?4,"Writes completed",!
  quit
retryEPIPE
  quit:$zstatus'["ENO32"
  use $p
  write "...Caught on try ",try,", write ",i,"... ",$zstatus,!
  set $ecode=""
  do readfrompipe(.pipe,.piperr)
  quit
retryEAGAIN
  quit:$zstatus'["ENO11"
  use $p
  write "...Failed to perform non-blocked writes... Retrying write # ",$increment(i,-1),!
  set $ecode=""
  hang 1+$random(5)
  use pipe
  quit

This example demonstrates how to handle PIPE device errors, whether with the device itself or from programs inside the PIPE device.

Example: 

sh> mumps -run pipexample induceEAGAIN
The active device is pipe OPEN PIPE SHELL="/bin/bash" COMMAND="$gtm_dist/mumps -run induceEAGAIN^pipexample" STDERR="piperr" 
$ZSTATUS="11,pipexample+9^pipexample,%SYSTEM-E-ENO11, Resource temporarily unavailable"
        
sh> mumps -run retry^pipexample induceEAGAIN
Try 0   pipe OPEN PIPE SHELL="/bin/bash" COMMAND="$gtm_dist/mumps -run induceEAGAIN^pipexample 0" STDERR="piperr"
...Failed to perform non-blocked writes... Retrying write # 54
...Failed to perform non-blocked writes... Retrying write # 63
...Failed to perform non-blocked writes... Retrying write # 69
...Failed to perform non-blocked writes... Retrying write # 78
    Writes completed

This example demonstrates handling WRITE errors, like ENO11 or EAGAIN, that do not terminate the PIPE device. The PIPE device does non-blocking writes. If a process tries to WRITE to a full PIPE and the WRITE would block, the device implicitly tries to complete the operation up to a default of 10 times. GT.M sleeps 100 micro seconds between each retry. When dealing with programs that can take a while to process input, it's a good idea to either schedule a delay between WRITEs or come up with a mechanism to back off the WRITEs when the buffer fills up.

sh> mumps -run pipexample induceEPIPE
The active device is pipe OPEN PIPE SHELL="/bin/bash" COMMAND="$gtm_dist/mumps -run induceEPIPE^pipexample" STDERR="piperr" 
    stdout:My PID is 12808
    stderr:%GTM-F-FORCEDHALT, Image HALTed by MUPIP STOP
$ZSTATUS="32,pipexample+9^pipexample,%SYSTEM-E-ENO32, Broken pipe"
        
sh> mumps -run retry^pipexample induceEPIPE
Try 0   pipe OPEN PIPE SHELL="/bin/bash" COMMAND="$gtm_dist/mumps -run induceEPIPE^pipexample 0" STDERR="piperr" 
...Caught on try 0, write 49... 32,retry+13^pipexample,%SYSTEM-E-ENO32, Broken pipe
    stdout:My PID is 16252
    stderr:%GTM-F-FORCEDHALT, Image HALTed by MUPIP STOP
Try 1   pipe OPEN PIPE SHELL="/bin/bash" COMMAND="$gtm_dist/mumps -run induceEPIPE^pipexample 1" STDERR="piperr" 
...Caught on try 1, write 697... 32,retry+13^pipexample,%SYSTEM-E-ENO32, Broken pipe
    stdout:My PID is 16403
    stdout:$ZSTATUS="150373210,induceEPIPE+5^pipexample,%GTM-E-DIVZERO, Attempt to divide by zero"
Try 2   pipe OPEN PIPE SHELL="/bin/bash" COMMAND="$gtm_dist/mumps -run induceEPIPE^pipexample 2" STDERR="piperr" 
    Writes completed

This example demonstrates how to create a separate STDERR pipe device from which to read the STDERR output of the program(s) inside the pipe. Reading the STDERR is important when dealing with failures from Unix programs. It is possible to read the errors without creating a STDERR pipe device, however the error messages are commingled with the output of the programs inside the pipe which could make diagnosis of the underlying problem harder. Notice that GT.M writes fatal errors, GTM-F types, to STDERR, but all others go to STDOUT.

Additionally, this example demonstrates handling errors that terminate the PIPE device. In this example, the PIPE device is terminated when a program inside the pipe terminates before reading all of the driving MUMPS program's output causing an EPIPE or ENO32, a broken pipe. In such a situation the MUMPS program must capture the error that caused the termination and respond accordingly. The program may need to call out to other programs to determine the status of a service it is using or to alert the operator of an error with an external program or service. To operate successfully, the program must recreate the pipe and retry the operation.

The following table summarizes the PIPE format deviceparameters.

DEVICE PARAMETER

CMD

DESCRIPTION

[NO]FIXED

O

Controls whether records have fixed length

RECORDSIZE=intexpr

O

Specifies the maximum record size.

VARIABLE

O

Controls whether records have variable length.

[Z]WIDTH=intexpr

U

Sets the device's logical record size and enables WRAP.

[Z][NO]WRAP

O/U

Controls the handling of records longer than the device width.

The following table summarizes PIPE access deviceparamters.

COMMAND=string

o

Specifies the command string to execut in a created process for the PIPE device. GT.M uses the default searching mechanism of the UNIX shell for creating the process and initiating its command(s).

SHELL=string

o

Specifies the path to a shell to be used instead of the default shell

STDERR=string

o

Specifies a device handle for a return pipe to which the created process writes any standard error output. The GT.M process can USE, READ, and CLOSE it, but cannot WRITE to it. When the GT.M process CLOSEs the PIPE device, the PIPE device CLOSEs STDERR, if still OPEN.

WRITEONLY

o

Specifies that the GT.M process may only WRITE to the created process via the PIPE device.

READONLY

o

Specifies that the GT.M process may only READ from the created process via the PIPE device. Output from both the standard output and the standard error output of the created process is available unless STDERR is specified.

PARSE

o

Specifies that GT.M parse the COMMAND and issue an OPEN exception for any invalid command.

INDEPENDENT

o

Specifies that the created process continues to execute after the PIPE device is CLOSEd.

Prev Up Next
 Home