The functions in programs increment and decrement are now available to GT.M through the shareable library libcrement.sl or libcrement.so, or though the DLL as libcrement.dll, depending on the specific platform. The suffix .sl is used throughout the following examples to represent .sl, .so, or .dll. Be sure to use the appropriate suffix for your platform.
GT.M uses an "external call table" to map the typeless data of M into the typed data of C, and vice versa. The external call table has a first line containing the pathname of the shareable library file followed by one or more specification lines in the following format:
entryref: return-value routine-name (parameter, parameter, ... ) [: SIGSAFE]
The optional case-insensitive keyword SIGSAFE following the parameter list specifies the external call does not create its own signal handlers; this allows GT.M to avoid burdensome signal handler coordination for the external call. By default, GT.M saves and restores signal setups for external calls.
entryref is an M entryref, return-value is gtm_long_t, gtm_status_t, or void, and parameters are in the format:
direction:type [num]
where [num] indicates a pre-allocation value explained later in this chapter.
Legal directions are I, O, or IO for input, output, or input/output, respectively.
The following table describes the legal types defined in the C header file $gtm_dist/gtmxc_types.h:
Type : Usage
void: Specifies that the function does not return a value.
gtm_status_t : Type int. If the function returns zero (0), then the call was successful. If it returns a non-zero value, GT.M will signal an error upon returning to M.
gtm_long_t : 32-bit signed integer on 32-bit platforms and 64-bit signed integer on 64-bit platforms.
gtm_ulong_t : 32-bit unsigned integer on 32-bit platforms and 64-bit signed integer on 64-bit platforms.
gtm_long_t* : For passing a pointer to long [integers].
gtm_float_t* : For passing a pointer to floating point numbers.
gtm_double_t* : Same as above, but double precision.
gtm_char_t*: For passing a "C" style string - null terminated.
gtm_char_t** : For passing a pointer to a "C" style string.
gtm_string_t* : For passing a structure in the form {int length;char *address}. Useful for moving blocks of memory to or from GT.M.
gtm_pointertofunc_t : For passing callback function pointers. For details see “Callback Mechanism”.
![[Note]](images/note.jpg) |
Note |
|---|---|
|
If an external call's function argument is defined in the external call table, GT.M allows invoking that function without specifying a value of the argument. All non-trailing and output-only arguments arguments which do not specify a value translate to the following default values in C:
In the mathpak package example, the following invocation translate inval to the default value, that is, 0. GTM>do &mathpak.increment(,.outval) If an external call's function argument is defined in the external call table and that function is invoked without specifying the argument, ensure that the external call function appropriately handles the missing argument. As a good programming practice, always ensure that count of arguments defined in the external call table matches the function invocation. To protect the process, GT.M turns any return values containing a null pointer to an empty string value and, for the first occurrence in a process, sends one XCRETNULLREF syslog message. If an external call sets a gtm_string length to a negative value, to protect the process, GT.M turns any return with a negative length to an empty string value and, for the first occurrence in a process, sends one XCCONVERT syslog message. |
gtmxc_types.h also includes definitions for the following entry points exported from libgtmshr:
void gtm_hiber_start(gtm_uint_t mssleep); void gtm_hiber_start_wait_any(gtm_uint_t mssleep) void gtm_start_timer(gtm_tid_t tid, gtm_int_t time_to_expir, void (*handler)(), gtm_int_t hdata_len, void \*hdata); void gtm_cancel_timer(gtm_tid_t tid);
where:
mssleep - milliseconds to sleep
tid - unique timer id value
time_to_expir - milliseconds until timer drives given handler
handler - function pointer to handler to be driven
hdata_len - 0 or length of data to pass to handler as a parameter
hdata - NULL or address of data to pass to handler as a parameter
gtm_hiber_start() always sleeps until the time expires; gtm_hiber_start_wait_any() sleeps until the time expires or an interrupt by any signal (including another timer). gtm_start_timer() starts a timer but returns immediately (no sleeping) and drives the given handler when time expires unless the timer is canceled.
![[Important]](images/important.jpg) |
Important |
|---|---|
GT.M continues to support xc_* equivalent types of gtm_* for upward compatibility. gtmxc_types.h explicitly marks the xc_* equivalent types as deprecated. |
Parameter-types that interface GT.M with non-M code using C calling conventions must match the data-types on their target platforms. Note that most addresses on 64-bit platforms are 8 bytes long and require 8 byte alignment in structures whereas all addresses on 32-bit platforms are 4 bytes long and require 4-byte alignment in structures.
Though strings with embedded zeroes are sent as input to external routines, embedded zeroes in output (or return value) strings of type gtm_char_t may cause string truncation because they are treated as terminator.
If your interface uses gtm_long_t or gtm_ulong_t types but your interface code uses int or signed int types, failure to revise the types so they match on a 64-bit platform will cause the code to fail in unpleasant, potentially dangerous and hard to diagnose ways.
The first parameter of each called routine is an int (for example, int argc in decrement.c and increment.c) that specifies the number of parameters passed. This parameter is implicit and only appears in the called routine. It does not appear in the call table specification, or in the M invocation. If there are no explicit parameters, the call table specification will have a zero (0) value because this value does not include itself in the count. If there are fewer actual parameters than formal parameters, the call is determined from the parameters specified by the values supplied by the M program. The remaining parameters are undefined. If there are more actual parameters than formal parameters, GT.M reports an error.
There may be only a single occurrence of the type gtm_status_t for each entryref.
To support Database Encryption, GT.M provides a reference implementation which resides in $gtm_dist/plugin/gtmcrypt.
The reference implementation includes:
A $gtm_dist/plugin/gtmcrypt sub-directory with all source files and scripts. The scripts include those needed to build/install libgtmcrypt.so and "helper" scripts, for example, add_db_key.sh (see below).
The plugin interface that GT.M expects is defined in gtmcrypt_interface.h. Never modify this file - it defines the interface that the plugin must provide.
$gtm_dist/plugin/libgtmcrypt.so is the shared library containing the executables which is dynamically linked by GT.M and which in turn calls the encryption packages. If the $gtm_dist/utf8 directory exists, then it should contain a symbolic link to ../plugin.
Source code is provided in the file $gtm_dist/plugin/gtmcrypt/source.tar which includes build.sh and install.sh scripts to respectively compile and install libgtmcrypt.so from the source code.
To support the implementation of a reference implementation, GT.M provides additional C structure types (in the gtmxc_types.h file):
gtmcrypt_key_t - a datatype that is a handle to a key. The GT.M database engine itself does not manipulate keys. The plug-in keeps the keys, and provides handles to keys that the GT.M database engine uses to refer to keys.
xc_fileid_ptr_t - a pointer to a structure maintained by GT.M to uniquely identify a file. Note that a file may have multiple names - not only as a consequence of absolute and relative path names, but also because of symbolic links and also because a file system can be mounted at more than one place in the file name hierarchy. GT.M needs to be able to uniquely identify files.
Although not required to be used by a customized plugin implementation, GT.M provides (and the reference implementation uses) the following functions for uniquely identifying files:
xc_status_t gtm_filename_to_id(xc_string_t *filename, xc_fileid_ptr_t *fileid) - function that takes a file name and provides the file id structure for that file.
xc_status_t gtm_is_file_identical(xc_fileid_ptr_t fileid1, xc_fileid_ptr_t fileid2) - function that determines whether two file ids map to the same file.
gtm_xcfileid_free(xc_fileid_ptr_t fileid) - function to release a file id structure.
Mumps, MUPIP and DSE processes dynamically link to the plugin interface functions that reside in the shared library. The functions serve as software "shims" to interface with an encryption library such as libmcrypt or libgpgme / libgcrypt.
The plugin interface functions are:
gtmcrypt_init()
gtmcrypt_getkey_by_name()
gtmcrypt_getkey_by_hash()
gtmcrypt_hash_gen()
gtmcrypt_encode()
gtmcrypt_decode()
gtmcrypt_close()
and gtmcrypt_strerror()
A GT.M database consists of multiple database files, each of which has its own encryption key, although you can use the same key for multiple files. Thus, the gtmcrypt* functions are capable of managing multiple keys for multiple database files. Prototypes for these functions are in gtmcrypt_interface.h.
The core plugin interface functions, all of which return a value of type gtm_status_t are:
gtmcrypt_init() performs initialization. If the environment variable $gtm_passwd exists and has an empty string value, GT.M calls gtmcrypt_init() before the first M program is loaded; otherwise it calls gtmcrypt_init() when it attempts the first operation on an encrypted database file.
Generally, gtmcrypt_getkey_by_hash or, for MUPIP CREATE, gtmcrypt_getkey_by_name perform key acquisition, and place the keys where gtmcrypt_decode() and gtmcrypt_encode() can find them when they are called.
Whenever GT.M needs to decode a block of bytes, it calls gtmcrypt_decode() to decode the encrypted data. At the level at which GT.M database encryption operates, it does not matter what the data is - numeric data, string data whether in M or UTF-8 mode and whether or not modified by a collation algorithm. Encryption and decryption simply operate on a series of bytes.
Whenever GT.M needs to encode a block of bytes, it calls gtmcrypt_encode() to encode the data.
If encryption has been used (if gtmcrypt_init() was previously called and returned success), GT.M calls gtmcrypt_close() at process exit and before generating a core file. gtmcrypt_close() must erase keys in memory to ensure that no cleartext keys are visible in the core file.
More detailed descriptions follow.
gtmcrypt_key_t *gtmcrypt_getkey_by_name(gtm_string_t *filename) - MUPIP CREATE uses this function to get the key for a database file. This function searches for the given filename in the memory key ring and returns a handle to its symmetric cipher key. If there is more than one entry for the given filename , the reference implementation returns the entry matching the last occurrence of that filename in the master key file.
gtm_status_t gtmcrypt_hash_gen(gtmcrypt_key_t *key, gtm_string_t *hash) - MUPIP CREATE uses this function to generate a hash from the key then copies that hash into the database file header. The first parameter is a handle to the key and the second parameter points to 256 byte buffer. In the event the hash algorithm used provides hashes smaller than 256 bytes, gtmcrypt_hash_gen() must fill any unused space in the 256 byte buffer with zeros.
gtmcrypt_key_t *gtmcrypt_getkey_by_hash(gtm_string_t *hash) - GT.M uses this function at database file open time to obtain the correct key using its hash from the database file header. This function searches for the given hash in the memory key ring and returns a handle to the matching symmetric cipher key. MUPIP LOAD, MUPIP RESTORE, MUPIP EXTRACT, MUPIP JOURNAL and MUPIP BACKUP -BYTESTREAM all use this to find keys corresponding to the current or prior databases from which the files they use for input were derived.
gtm_status_t gtmcrypt_encode(gtmcrypt_key_t *key, gtm_string_t *inbuf, gtm_string_t *outbuf) and gtm_status_t gtmcrypt_decode(gtmcrypt_key_t *key, gtm_string_t *inbuf, gtm_string_t *outbuf)- GT.M uses these functions to encode and decode data. The first parameter is a handle to the symmetric cipher key, the second a pointer to the block of data to encode or decode, and the third a pointer to the resulting block of encoded or decoded data. Using the appropriate key (same key for a symmetric cipher), gtmcrypt_decode() must be able to decode any data buffer encoded by gtmcrypt_encode(), otherwise the encrypted data is rendered unrecoverable.7 As discussed earlier, GT.M requires the encrypted and cleartext versions of a string to have the same length.
char *gtmcrypt_strerror() - GT.M uses this function to retrieve addtional error context from the plug-in after the plug-in returns an error status. This function returns a pointer to additional text related to the last error that occurred. GT.M displays this text as part of an error report. In a case where an error has no additional context or description, this function returns a null string.
The complete source code for reference implementations of these functions is provided, licensed under the same terms as GT.M. You are at liberty to modify them to suit your specific GT.M database encryption needs. Check your GT.M license if you wish to consider redistributing your changes to others.
For more information and examples, refer to the Database Encryption Technical Bulletin.
The definition of parameters passed by reference with direction output can include specification of a pre-allocation value. This is the number of units of memory that the user wants GT.M to allocate before passing the parameter to the external routine. For example, in the case of type gtm_char_t *, the pre-allocation value would be the number of bytes to be allocated before the call to the external routine.
Specification of a pre-allocation value should follow these rules:
Pre-allocation is an unsigned integer value specifying the number of bytes to be allocated on the system heap with a pointer passed into the external call.
Pre-allocating on a type with a direction of input or input/output results in a GT.M error.
Pre-allocation is meaningful only on types gtm_char_t * and gtm_string_t *. On all other types the pre-allocation value specified will be ignored and the parameter will be allocated a default value for that type. With gtm_string_t * arguments make sure to set the 'length' field appropriately before returning control to GT.M. On return from the external call, GT.M uses the value in the length field as the length of the returned value, in bytes.
If the user does not specify any value, then the default pre-allocation value would be assigned to the parameter.
Specification of pre-allocation for "scalar" types (parameters which are passed by value) is an error.
|
Important |
|---|---|
Pre-allocation is optional for all output-only parameters except gtm_string_t * and gtm_char_t *. Pre-allocation yields better management of memory for the external call. When an external call exceeds its specified preallocation (gtm_string_t * or gtm_char_t * output), GT.M produces the EXCEEDSPREALLOC error. In the case the user allocates space for the character pointer inside a gtm_string_t * type output parameter, a length field longer than the specified preallocate size for the output parameter does not cause an EXCEEDSPREALLOC error. |
GT.M exposes certain functions that are internal to the GT.M runtime library for the external calls via a callback mechanism. While making an external call, GT.M populates and exposes a table of function pointers containing addresses to call-back functions.
|
Index |
Function |
Argument |
Type |
Description |
|---|---|---|---|---|
|
0 |
hiber_start |
sleep for a specified time |
||
|
slp_time |
integer |
milliseconds to sleep |
||
|
1 |
hiber_start_wait_any |
sleep for a specified time or until any interrupt, whichever comes first |
||
|
slp_time |
integer |
milliseconds to sleep |
||
|
2 |
start_timer |
start a timer and invoke a handler function when the timer expires |
||
|
tid |
integer |
unique user specified identifier for this timer |
||
|
time_to_expire |
integer |
milliseconds before handler is invoked |
||
|
handler |
pointer to function |
specifies the entry of the handler function to invoke |
||
|
hlen |
integer |
length of data to be passed via the hdata argument |
||
|
hdata |
pointer to char |
data (if any) to pass to the handler function |
||
|
3 |
cancel_timer |
stop a timer previously started with start_timer(), if it has not yet expired |
||
|
tid |
integer |
unique user specified identifier of the timer to cancel |
||
|
4 |
gtm_malloc |
allocates process memory from the heap |
||
|
<return-value> |
pointer to void |
address of the allocated space |
||
|
space_needed |
32-bit platforms: 32-bit unsigned integer 64-bit platforms: 64-bit unsigned integer |
bytes of space to allocate. This has the same signature as the system malloc() call. |
||
|
5 |
gtm_free |
return memory previously allocated with gtm_malloc() |
||
|
free_address |
pointer to void |
address of the previously allocated space |
The external routine can access and invoke a call-back function in any of the following mechanisms:
While making an external call, GT.M sets the environment variable GTM_CALLIN_START to point to a string containing the start address (decimal integer value) of the table described above. The external routine needs to read this environment variable, convert the string into an integer value and should index into the appropriate entry to call the appropriate GT.M function.
GT.M also provides an input-only parameter type gtm_pointertofunc_t that can be used to obtain call-back function pointers via parameters in the external routine. If a parameter is specified as I:gtm_pointertofunc_t and if a numeric value (between 0-5) is passed for this parameter in M, GT.M interprets this value as the index into the callback table and passes the appropriate callback function pointer to the external routine.
|
Note |
|---|---|
FIS strongly discourages the use of signals, especially SIGALARM, in user written C functions. GT.M assumes that it has complete control over any signals that occur and depends on that behavior for recovery if anything should go wrong. The use of exposed timer APIs should be considered for timer needs. |
Since both GT.M runtime environment and the external C functions execute in the same process space, the following restrictions apply to the external functions:
GT.M is designed to use signals and has signal handlers that must function for GT.M to operate properly. The timer related call-backs should be used in place of any library or system call which uses SIGALRM such as sleep(). Use of signals by external call code may cause GT.M to fail.
Use of the GT.M provided malloc and free, creates an integrated heap management system, which has a number of debugging tools. FIS recommends the usage of gtm_malloc/gtm_free in the external functions that provides better debugging capability in case memory management problems occur with external calls.
Use of exit system call in external functions is strongly discouraged. Since GT.M uses exit handlers to properly shutdown runtime environment and any active resources, the system call _exit should never be used in external functions.
GT.M uses timer signals so often that the likelihood of a system call being interrupted is high. So, all system calls in the external program can return EINTR if interrupted by a signal.
Handler functions invoked with start_timer must not invoke services that are identified by the Operating System documentation as unsafe for signal handlers (or not identified as safe) - consult the system documentation or man pages for this information. Such services cause non-deterministic failures when they are interrupted by a function that then attempts to call them, wrongly assuming they are reentrant.
foo: void bar (I:gtm_float_t*, O:gtm_float_t*)
There is one external call table for each package. The environment variable "GTMXC" must name the external call table file for the default package. External call table files for packages other than the default must be identified by environment variables of the form "GTMXC_name".
The first of the external call tables is the location of the shareable library. The location can include environment variable names.
Example:
% echo $GTMXC_mathpak
/user/joe/mathpak.xc
% echo lib /usr/
% cat mathpak.xc
$lib/mathpak.sl
exp: gtm_status_t xexp(I:gtm_float_t*, O:gtm_float_t*)
% cat exp.c
...
int xexp(count, invar, outvar)
int count;
float *invar;
float *outvar;
{
...
}
% gtm
...
GTM>d &mathpak.exp(inval,.outval)
GTM>
Example : For preallocation:
% echo $GTMXC_extcall
/usr/joe/extcall.xc
% cat extcall.xc
/usr/lib/extcall.sl
prealloc: void gtm_pre_alloc_a(O:gtm_char_t *[12])
% cat extcall.c
#include <stdio.h>
#include <string.h>
#include "gtmxc_types.h"
void gtm_pre_alloc_a (int count, char *arg_prealloca)
{
strcpy(arg_prealloca, "New Message");
return;
}
Example : for call-back mechanism
% echo $GTMXC
/usr/joe/callback.xc
% cat /usr/joe/callback.xc
$MYLIB/callback.sl
init: void init_callbacks()
tstslp: void tst_sleep(I:gtm_long_t)
strtmr: void start_timer(I:gtm_long_t, I:gtm_long_t)
% cat /usr/joe/callback.c
#include <stdio.h>
#include <stdlib.h>
#include "gtmxc_types.h"
void **functable;
void (*sleep_uninterrupted)(int );
void (*sleep_interrupted)(int );
void (*setup_timer)(int , int , void (*)() , int , char *);
void (*cancel_timer)(int );
void* (*malloc_fn)(int);
void (*free_fn)(void*);
void init_callbacks (int count)
{
char *start_address;
start_address = (char *)getenv("GTM_CALLIN_START");
if (start_address == (char *)0)
{
fprintf(stderr,"GTM_CALLIN_START is not set\n");
return;
}
functable = (void **)atoi(start_address);
if (functable == (void **)0)
{
perror("atoi : ");
fprintf(stderr,"addresses defined by GTM_CALLIN_START not a number\n");
return;
}
sleep_uninterrupted = (void (*)(int )) functable[0];
sleep_interrupted = (void (*)(int )) functable[1];
setup_timer = (void (*)(int , int, void (*)(), int, char *)) functable[2];
cancel_timer = (void (*)(int )) functable[3];
malloc_fn = (void* (*)(int)) functable[4];
free_fn = (void (*)(void*)) functable[5];
return;
}
void sleep (int count, int time)
{
(*sleep_uninterrupted)(time);
}
void timer_handler ()
{
fprintf(stderr,"Timer Handler called\n");
/* Do something */
}
void start_timer (int count, int time_to_int, int time_to_sleep)
{
(*setup_timer)((int )start_timer, time_to_int, timer_handler, 0, 0);
return;
}
void* xmalloc (int count)
{
return (*malloc_fn)(count);
}
void xfree(void* ptr)
{
(*free_fn)(ptr);
}
Example:gtm_malloc/gtm_free callbacks using gtm_pointertofunc_t
% echo $GTMXC
/usr/joe/callback.xc
% cat /usr/joe/callback.xc
/usr/lib/callback.sl
init: void init_callbacks(I:gtm_pointertofunc_t, I:gtm_pointertofunc_t)
% gtm
GTM> do &.init(4,5)
GTM>
% cat /usr/joe/callback.c
#include <stdio.h>
#include <stdlib.h>
#include "gtmxc_types.h"
void* (*malloc_fn)(int);
void (*free_fn)(void*);
void init_callbacks(int count, void* (*m)(int), void (*f)(void*))
{
malloc_fn = m;
free_fn = f;
}