Using Sequential Files

Using Sequential Files
Prev	Chapter 9. Input/Output Processing	Next

Setting Sequential File Characteristics

The ANSI standard specifies that when a process CLOSEs and then reOPENs a device, GT.M restores any characteristics not explicitly specified with deviceparameters to the values they had prior to the last CLOSE. However, because it is difficult for a large menu-driven application to ensure the previous OPEN state, GT.M always sets unspecified sequential file characteristics to their default value on OPEN. This approach also reduces potential memory overhead imposed by OPENing and CLOSEing a large number of sequential files during the life of a process.

GT.M does not restrict multiple OPEN commands. However, if a file is already open, GT.M ignores attempts to modify sequential file OPEN characteristics, except for RECORDSIZE and for deviceparameters that also exist for USE.

Sequential files can be READONLY, or read/write (NOREADONLY).

Sequential files can be composed of either FIXED or VARIABLE (NOFIXED) length records. By default, records have VARIABLE length.

UNIX enforces its standard security when GT.M OPENs a sequential file. This includes any directory access required to locate or create the file. If you are unable to OPEN a file, contact your system manager.

Sequential File Pointers

Sequential file I/O operations use a construct called a file pointer. The file pointer logically identifies the next record to read or write. OPEN commands position the file pointer at the beginning of the file (REWIND) or at the end-of-file (APPEND). APPEND cannot reposition a file currently open. Because the position of each record depends on the previous record, a WRITE destroys the ability to reliably position the file pointer to subsequent records in a file. Therefore, by default (NOTRUNCATE), GT.M permits WRITEs only when the file pointer is positioned at the end of the file.

A file that has been previously created and contains data that should be retained can also be opened with the device parameter APPEND.

If a device has TRUNCATE enabled, a WRITE issued when the file pointer is not at the end of the file causes all contents after the current file pointer to be discarded. This effectively moves the end of the file to the current position and permits the WRITE.

Line Terminators

LF ($CHAR(10)) terminates the logical record for all M mode sequential files, TRM, PIPE, and FIFO. For non FIXED format sequential files and terminal devices for which character set is not M, all the standard Unicode® line terminators terminate the logical record. These are U+000A (LF), U+0000D (CR), U+000D followed by U+000A (CRLF), U+0085 (NEL), U+000C (FF), U+2028 (LS) and U+2029 (PS).

READ/WRITE Operations

The following table describes all READ and WRITE operations for STREAM, VARIABLE, and FIXED format sequential files having automatic record termination enabled (WRAP) or disabled (NOWRAP).

Command	WRAP or NOWRAP	STREAM or VARIABLE format file behavior		FIXED format file behavior
READ format or WRITE or WRITE *	WRAP	Write the entire argument, but anytime $X is about to exceed WIDTH: insert a <LF> character, set $X to 0, increment $Y		Similar to VARIABLE but no <LF>
READ format or WRITE or WRITE *	NOWRAP	Update $X based on STREAM or VARIABLE format as described below		Same as VARIABLE
READ format or WRITE or WRITE *	NOWRAP	STREAM: Write all of the argument with no truncation nor with a line terminator being inserted. Add length of argument to $X.	VARIABLE ($X=WIDTH): Write up to WIDTH-$X characters. Write no more output to the device until a WRITE ! or a SET $X makes $X less than WIDTH.	Same as VARIABLE
READ or WRITE !	either	Write <LF>, set $X to 0, increment $Y		Write PAD bytes to bring the current record to WIDTH
WRITE #	either	Write <FF>,<LF> or <FF>, set $X and $Y to 0. See the documentation for the USE and OPEN commands to understand how the [NO]FFLF device parameter and the gtm_nofflf environment variable affects the operation of "WRITE #". If the device parameter is unused, and the environment variable is unset, the default behavior of "WRITE #" is to write <FF>,<LF>.		Write PAD bytes to bring the current record to WIDTH, then a <FF> followed by WIDTH-1 PAD bytes
CLOSE	either	After a WRITE, if $X > 0, Write <LF>		After a WRITE, if $X >0, perform an implicit "WRITE !" adding PAD bytes to create a full record. If you need to avoid trailing PAD bytes set $X to 0 before closing a FIXED format file.
READ X	either	Return characters up to $X=WIDTH, or until encountering an <LF> or EOF. If <LF> encountered, set $X to 0, increment $Y		Return WIDTH characters; no maintenance of $X and $Y, except that EOF increments $Y
READ X#len	either	Return characters up to the first of $X=WIDTH or len characters, or encountering a <LF> or EOF; if up to len characters or EOF update $X, otherwise set $X to 0 and increment $Y		Return MIN(WIDTH, len) characters; no maintenance of $X and $Y, except that EOF increments $Y
READ *X	either	Return the code for one character and increment $X, if WIDTH=$X or <LF> encountered, set $X=0, increment $Y; if EOF return -1		Return the code for one character, if EOF return -1; no maintenance of $X and $Y, except that EOF increments $Y

Note

EOF == end-of-file; <FF>== ASCII form feed; <LF> == ASCII line feed;
In M mode, and by default in UTF-8 mode PAD == <SP> == ASCII space.
"READ format" in this table means READ ? or READ <strlit>
A change to WIDTH implicitly sets WRAP unless NOWRAP follows in the deviceparameter list
In VARIABLE and STREAM mode, READ (except for READ *) never returns <LF> characters
In M mode, the last setting of RECORDSIZE or WIDTH for the device determines WIDTH
In M Mode, a WRITE to a sequential device after setting $X to a value greater than the device WIDTH or a reducing WIDTH to less than the current $X acts as if the first character caused $X to exceed the WIDTH induces an immediate WRAP, if WRAP is enabled
In UTF-8 mode, RECORDSIZE is in bytes and WIDTH is in characters and the smaller acts as the WIDTH limit in the table.
In UTF-8 mode, FIXED mode writes <SP> to the RECORDSIZE when the next character won't fit.
In UTF-8 mode, all READ forms do not return trailing PAD characters.
In UTF-8 mode, all characters returned by all forms of FIXED mode READ are from a single record.
WRITE for a Sequential Disk (SD) device works at the current file position, whether attained with APPEND, REWIND or SEEK.
GT.M manages any BOM for UTF mode files by ensuring they are at the beginning of the file and produces a BOMMISMATCH error for an attempt to change the byte-ordering on OPEN for an existing file.
An attempt to OPEN a non-zero length file WRITEONLY without either NEWVERSION or TRUNCATE in UTF mode produces an OPENDEVFAIL due to the fact that any existing BOM information cannot be verified.
GT.M SD encryption, because of the state information associated with encryption processing, requires encrypted files to be WRITEn or READ from the beginning rather than from an arbitrary position.
HEREDOCs in shell scripts that drive input to GT.M present to GT.M as SD files, and by default terminate with a success - zero (0) status unless GT.M terminates with a ZHALT that supplies an alternative status.

Writing Binary Files

To write a binary data file, open it with FIXED:WRAP:CHSET="M" and set $X to zero before the WRITE to avoid filling the last record with spaces (the default PAD byte value).

	Note
	With CHSET not "M", FIXED has a different definition. Each record is really the same number of bytes as specified by RECORDSIZE. Padding bytes are added as needed to each record.

Sequential File Deviceparameter Summary

The following tables provide a brief summary of deviceparameters for sequential files grouped into related areas. For more detailed information, refer to “Open”, “Use”, and “Close”.

Error Processing Deviceparameters
DEVICEPARAMETER	COMMAND	COMMENT
EXCEPTION=expr	O/U/C	Controls device-specific error handling.

File Pointer Positioning Deviceparameters
DEVICEPARAMETER	COMMAND	COMMENT
APPEND	O	Positions file pointer at EOF.
REWIND	O/U/C	Positions file pointer at start of the file.
SEEK=strexpr	O/U	Positions the current file pointer to the location specified in strexpr. The format of strexpr is a string of the form "[+\|-]integer" where unsigned value specifies an offset from the beginning of the file, and an explicitly signed value specifies an offset relative to the current file position. For STREAM or VARIABLE format, the positive intexpr after any sign is a byte offset, while for a FIXED format, it is a record offset. In order to deal with the possible presence of a Byte Order Marker (BOM), SEEK for a FIXED format file written in a UTF character set must follow at least one prior READ since the device was created.

File Format Deviceparameters
DEVICEPARAMETERS	COMMAND	COMMENT
[NO]FIXED	O	Controls whether records have fixed length.
[Z]LENGTH=intexpr	U	Controls virtual page length.
RECORDSIZE=intexpr	O	Specifies maximum record size.
STREAM	O	Specifies the STREAM format.
VARIABLE	O	Controls whether records have variable length.
[Z]WIDTH=intexpr	U	Controls maximum width of an output line.
[Z][NO]WRAP	O/U	Controls handling of records longer than device width.

File Access Deviceparameters
DEVICEPARAMETER	COMMAND	COMMENT
DELETE	C	Specifies file be deleted by CLOSE.
GROUP=expr	O/C	Specifies file permissions for other users in the owner's group.
NEWVERSION	O	Specifies GT.M create a new version of file.
OWNER=expr	O/C	Specifies file permissions for the owner of file.
[NO]READONLY	O	Controls read-only file access.
RENAME=expr	C	Specifies CLOSE rename a disk file with name specified by expression.
REPLACE=expr	C	Specifies CLOSE replace(overwrite an existing file if necessary) the name of a disk file with the name specified by the expression.
SYSTEM=expr	O/C	Specifies file permissions for the owner of the file (same as OWNER).
[NO]TRUNCATE	O/U	Controls overwriting of existing data in file.
UIC=expr	O/C	Specifies file's owner ID.
WORLD=expr	O/C	Specifies file permissions for users not in the owner's group.

O: Applies to the OPEN command

U: Applies to the USE command

C: Applies to the CLOSE command

Sequential File Examples

This section contains a few brief examples of GT.M sequential file handling.

Example:

GTM>do ^FREAD
FREAD;
 zprint ^FREAD 
 read "File > ",sd
 set retry=0
 set $ztrap="BADAGAIN"
 open sd:(readonly:exception="do BADOPEN")
 use sd:exception="goto EOF"
 for  use sd read x use $principal write x,!
EOF;
 if '$zeof zmessage +$zstatus
 close sd
 quit
BADOPEN;
 set retry=retry+1 
 if retry=2 open sd
 if retry=4 halt
 if $piece($zstatus,",",1)=2 do  
 . write !,"The file ",sd," does not exist. Retrying in about 2 seconds ..."
 . hang 2.1
 . quit 
 if $piece($zstatus,",",1)=13 do  
 . write !,"The file ",sd," is not accessible. Retrying in about 3 seconds ..."
 . hang 3.1
 . quit
 quit
BADAGAIN;
 w !,"BADAGAIN",!
 
File >

This example asks for the name of the file and displays its contents. It OPENs that file as READONLY and specifies an EXCEPTION. The exception handler for the OPEN deals with file-not-found and file-access errors and retries the OPEN command on error. The first USE sets the EXCEPTION to handle end-of-file. The FOR loop reads the file one record at a time and transfers each record to the principal device. The GOTO in the EXCEPTION terminates the FOR loop. At label EOF, if $ZEOF is false, the code reissues the error that triggered the exception. Otherwise, the CLOSE releases the file.

Example:

GTM>do ^formatACCT
formatACCT;
 zprint ^formatACCT; 
 set sd="temp.dat",acct=""
 open sd:newversion 
 use sd:width=132
 for  set acct=$order(^ACCT(acct)) quit:acct=""  do  
 . set rec=$$FORMAT(acct)
 . write:$y>55 #,hdr write !,rec
 close sd
 quit

This OPENs a NEWVERSION of file temp.dat. The FOR loop cycles through the ^ACCT global formatting (not shown in this code fragment) lines and writing them to the file. The FOR loop uses the argumentless DO construct to break a long line of code into more manageable blocks. The program writes a header record (set up in initialization and not shown in this code fragment) every 55 lines, because that is the application page length, allowing for top and bottom margins.