The GT.M Journaling facility has impact on a number of components:
DSE - used to examine current journaling configuration and historical statistics
GDE - used to set up default journaling characteristics for database regions
MUPIP JOURNAL - used to process journal files for recovery or analysis
MUPIP JOURNAL /ROLLBACK - used to recover databases when replication is enabled - relies on journal files
MUPIP BACKUP - used to make database backups - can initiate new journal files
MUPIP SET /JOURNAL - used to modify journaling characteristics, and to initiate new journal files
MUPIP SET /REPLICATION - used to modify replication characteristics - replication requires journaling
GT.M compiler - supports language features (with run-time system support) to examine journaling status and, when journaling is on, affect journaling behavior
GT.M run-time system - creates journal file entries; may create new journal files or turn off journaling under some circumstances;
In order to use journaling you should familiarize yourself with the tools. One critical choice is deciding whether and how to use journal fences. Another is how to manage the journal files in coordination with an appropriate database backup scheme. Whether or not to use replication affects these choices.
The following sections address these journaling topics in more detail.
The programming practice of fencing logical transactions protects database integrity during a system interruption. A logical transaction is a logical unit that is not complete unless all parts of the transaction are captured. For instance, the logical transaction "transfer funds between accounts" consists of a debit update to one account and a credit update to another account.
Establishing fences around a logical transaction assures that the transaction is committed as a unit, thereby avoiding logical inconsistencies. These logical inconsistencies, sometimes referred to as application-level database integrity problems, manifest themselves as run-time errors, inappropriate branching, and incorrect reports.
In GT.M, there are two methods of defining application transaction boundaries. One method is using the transaction commands TSTART and TCOMMIT. The other method uses the GT.M journal fencing commands ZTSTART and ZTCOMMIT.
TSTART or ZTSTART activate transaction fence control. Subsequent SETs and KILLs across all accessed regions are marked as belonging to a fenced transaction. The TCOMMIT or ZTCOMMIT commands respectively, close the fenced transaction and write the TCOMMIT and ZTCOMMIT journal records in all the journal files of all regions involved in the transaction.
If a TSTART has been processed, and another TSTART is encountered prior to a TCOMMIT, then the second TSTART does not cause another fence to be emitted to the journal file, instead increments a "transaction depth" counter accessible as $TLEVEL. Succeeding TCOMMIT directives cause decrementing of this transaction depth counter. The TCOMMIT fence is emitted to the journal file when the "transaction depth" returns to zero (0) (such occurrences are referred to as "nested" transactions or sub-transactions).
The maximum depth of a TP transaction nesting is 127 levels.
Similarly, if a ZTSTART has been processed, and another ZTSTART is encountered prior to a ZTCOMMIT, then the second ZTSTART does not cause another fence to be emitted to the journal file. Instead the second ZTSTART increments a "transaction depth" counter. Succeeding ZTCOMMIT directives cause decrementing of this transaction depth counter. The ZTCOMMIT fence is emitted to the journal file when the "transaction depth" returns to zero (0) (such occurrences are referred to as "nested" transactions or sub-transactions).
The maximum depth of a ZTSTART transaction nesting is 255 levels.
The command ZTCOMMIT numexpr where numexpr is greater than zero (0), is functionally equivalent to the code:
X "NEW X F X=1:1:numexpr ZTCOMMIT"
and the command ZTCOMMIT 0 is equivalent to the command:
ZTCOMMIT $VIEW("JNLTRANSACTION")
This essentially means, ZTCOMMIT 0 closes the transaction, and any and all open sub-transactions.
Negative values of numexpr are erroneous.
In case of TSTART/TCOMMIT, journal records for all fenced updates involved in the transaction are written to journal files of all regions only during the time of TCOMMIT, that is, only when the TCOMMIT journal records get written. While, in the case of ZTSTART/ZTCOMMIT, journal records are written as and when each SET or KILL is encountered in the fenced transaction.
While TSTART and TCOMMIT produce transactions with ACID properties (when certain conventions are followed), ZTSTART and ZTCOMMIT do not modify M semantics at all. These commands only generate fencing information in the journals. The M and Z transaction commands cannot be mixed in a single transaction. MUPIP JOURNAL uses the information provided by TSTART and TCOMMIT or ZTSTART and ZTCOMMIT, to restore the entire set of updates that comprise the transaction (unless specifically instructed to restore only part of the update).
Using TSTART and TCOMMIT or ZTSTART and ZTCOMMIT significantly improves the benefit of journaling. When recovering from journal files, MUPIP JOURNAL /RECOVER processing by default recovers either all updates within the transaction fences or none of them, and reports the latter case during recovery.
In MUPIP documentation, the term transaction means "database update".
You can fence some, all, or none of your application programs. When you program with fences, you can still force a recovery to ignore the fences by using additional qualifiers to MUPIP JOURNAL /RECOVER. The following lists advantages and disadvantages for fencing transactions.
Faster recovery
Minimum risk recovery
Databases recovered from journals that include fences do not require post-recovery checks and repairs for logical consistency
Must be programmed into the M code
If the application is already structured to minimize logical transaction breakage problems, inserting the fencing commands may be a largely mechanical task. In less structured applications, inserting fences immediately "inside" the M LOCKs associated with transactions may provide an excellent first approximation of proper fencing.
Fencing requires additional CPU resources and adds entries to the journal file(s)
Fencing may duplicate methods of recovery already established to address these issues
An application structured so that all information for each logical transaction is stored in a single global node (while other nodes hold only redundant information), permits rebuild programs to completely correct logical inconsistencies. With less restrictive designs, logical inconsistencies may be corrected manually or by using semi-automated techniques.
LOCKs and ZTSTART and ZTCOMMIT transaction fences have an important relationship. MUPIP JOURNAL /RECOVER assumes that fenced transactions are isolated and never performs a cascading roll-back. The term cascading roll-back describes the situation that occurs when dropping one transaction causes previous transactions to be sequentially dropped, until potentially all transactions are dropped. If an application violates this assumption, a JOURNAL /RECOVER may create a database with application-level integrity problems. M LOCKs ensure the isolation of a sequence of updates from interaction with any other updates. TSTART and TCOMMIT transaction fences implicitly exhibit the required isolation whether fences are used with or without associated LOCKs.
Example of a non-isolated transaction:
Process A Process B
--------- ---------
ZTS
S ^(acct)=^AMT(d,acct)+amt
ZTS
S ^(acct)=^AMT(d,acct)+amt
S (j,^(0))=^JNL(acct,0)+1
S ^JNL(acct,j)=stf
ZTC
S (j,^(0))=^JNL(acct,0)+1
-- failure --These two sets of transactions are not isolated from each other. After the system fails, Process A's transaction must be removed by a /RECOVER /BACKWARD or not replayed by a /RECOVER /FORWARD. However, if acct has the same value for both processes when Process B's transaction is recovered, ^AMT(d,acct) includes Process A's amt as well as Process B's amt.
Adding a LOCK to the program (e.g., L ^JNL(acct) just before the ZTSTART and a LOCK with no argument after the ZTCOMMIT,) isolates the transactions and prevents the two processes from overlapping. Because M does not treat the SET command as an atomic database operation, in the absence of TSTART/TCOMMIT commands, such LOCKs should exist independent of journaling considerations to protect the updates to ^AMT(d,acct) and ^JNL(0).