Handling Lists, Arrays, and Other Expressions

The Wolfram Symbolic Transfer Protocol (WSTP) allows you to exchange data of any type with external programs. For more common types of data, you simply need to give appropriate :ArgumentTypes: or :ReturnType: specifications in your WSTP template file.

Basic type specifications.

Here is the WSTP template for a function that takes a list of integers as its argument.

:Begin:
:Function:       h
:Pattern:        h[a_List]
:Arguments:      {a}
:ArgumentTypes:  {IntegerList}
:ReturnType:     Integer
:End:

Here is the C source code for the function. Note the extra argument alen which is used to pass the length of the list.

int h(int *a, long alen) {

   int i, tot=0;

   for(i=0; i<alen; i++)
      tot += a[i];

   return tot;
}

This installs an external program containing the specifications for the function h:

Install["hprog"]

This calls the external code:

h[{3,5,6}]

This does not match the pattern h[a_List] so does not call the external code:

h[67]

The pattern is matched, but the elements in the list are of the wrong type for the external code, so (RefLink[$Failed, paclet:ref/$Failed]) is returned:

h[{a,b,c}]

You can mix basic types of arguments in any way you want. Whenever you use IntegerList or RealList, however, you have to include an extra argument in your C program to represent the length of the list.

Here is an :ArgumentTypes: specification.

:ArgumentTypes:  {IntegerList, RealList, Integer}

Here is a possible corresponding C function declaration.

void f(int *a, long alen, double *b, long blen, int c)

Note that when a list is passed to a C program by WSTP its first element is assumed to be at position 0, as is standard in C, rather than at position 1, as is standard in the Wolfram Language.

In addition, following C standards, character strings specified by (RefLink[String, paclet:ref/String]) are passed as char* objects, terminated by ∖0 null bytes. "Portability of WSTP Programs" discusses how to handle special characters.

WSTP functions for sending data to the Wolfram Language.

When you use a WSTP template file, what (RefLink[mprep, paclet:ref/program/mprep]) and (RefLink[mcc, paclet:ref/program/mcc]) actually do is to create a C program that includes explicit calls to WSTP library functions. If you want to see an example of how to use the WSTP library functions directly, you can look at the source code of this program. Note when you use (RefLink[mcc, paclet:ref/program/mcc]), you typically need to give a -g option, otherwise the source code that is generated is automatically deleted.

If your external function just returns a single integer or floating-point number, then you can specify this just by giving (RefLink[Integer, paclet:ref/Integer]) or (RefLink[Real, paclet:ref/Real]) as the :ReturnType: in your WSTP template file. But because of the way memory allocation and deallocation work in C, you cannot directly give :ReturnType: specifications such as IntegerList or RealList. And instead, to return such structures, you must explicitly call WSTP library functions within your C program, and give (RefLink[Manual, paclet:ref/Manual]) as the :ReturnType: specification.

Here is the WSTP template for a function that takes an integer as an argument, and returns a list of the digits in its binary representation using explicit WSTP functions.

:Begin:
:Function:       bits
:Pattern:        bits[i_Integer]
:Arguments:      {i}
:ArgumentTypes:  {Integer}
:ReturnType:     Manual
:End:

The return type of the function is declared as void.

void bits(int i) {

   int a[32], k;

This puts values into the C array a.

for(k=0; k<32; k++) {
       a[k] = i%2;
       i >>= 1;
       if (i==0) break;
   }

   if (k<32) k++;

This sends k elements of the array a back to the Wolfram Language.

WSPutInteger32List(stdlink, a, k);
    return ;
}

This installs the program containing the external function bits:

Install["bitsprog"]

The external function now returns a list of bits:

bits[14]

If you declare an array in C as int a[(Subscript[n, 1])][(Subscript[n, 2])][(Subscript[n, 3])], then you can use (NonWLLink[WSPutInteger32Array(), paclet:ref/c/WSPutInteger32Array]) to send it to the Wolfram Language as a depth 3 list.

Here is a declaration for a 3-dimensional C array.

int a[8][16][100];

This sets up the array dims and initializes it to the dimensions of a.

int dims[] = {8, 16, 100};

This sends the 3-dimensional array a to the Wolfram Language, creating a depth 3 list.

WSPutInteger32Array(stdlink, a, dims, NULL, 3);

You can use WSTP functions to create absolutely any Wolfram Language expression. The basic idea is to call a sequence of WSTP functions that correspond directly to the (RefLink[FullForm, paclet:ref/FullForm]) representation of the Wolfram Language expression.

This sets up the Wolfram Language function (RefLink[Plus, paclet:ref/Plus]) with 2 arguments.

WSPutFunction(stdlink, "Plus", 2);

This specifies that the first argument is the integer 77.

WSPutInteger32(stdlink, 77);

And this specifies that the second argument is the symbol x.

WSPutSymbol(stdlink, "x");

In general, you first call (NonWLLink[WSPutFunction(), paclet:ref/c/WSPutFunction]), giving the head of the Wolfram Language function you want to create, and the number of arguments it has. Then you call other WSTP functions to fill in each of these arguments in turn. "Expressions" discusses the general structure of Wolfram Language expressions and the notion of heads.

This creates a Wolfram Language list with 2 elements.

WSPutFunction(stdlink, "List", 2);

The first element of the list is a list of 10 integers from the C array r.

WSPutInteger32List(stdlink, r, 10);

The second element of the main list is itself a list with 2 elements.

WSPutFunction(stdlink, "List", 2);

The first element of this sublist is a floating-point number.

WSPutReal64(stdlink, 4.5);

The second element is an integer.

WSPutInteger32(stdlink, 11);

(NonWLLink[WSPutInteger32Array(), paclet:ref/c/WSPutInteger32Array]) and (NonWLLink[WSPutReal64Array(), paclet:ref/c/WSPutReal64Array]) allow you to send arrays which are laid out in memory in the one-dimensional way that C pre-allocates them. But if you create arrays during the execution of a C program, it is more common to set them up as nested collections of pointers. You can send such arrays to the Wolfram Language by using a sequence of (NonWLLink[WSPutFunction(), paclet:ref/c/WSPutFunction]) calls, ending with an (NonWLLink[WSPutInteger32List(), paclet:ref/c/WSPutInteger32List]) call.

This declares a to be a nested list of lists of lists of integers.

int ***a;

This creates a Wolfram Language list with n1 elements.

WSPutFunction(stdlink, "List", n1);

for (i=0; i<n1; i++) {

This creates a sublist with n2 elements.

WSPutFunction(stdlink, "List", n2);

for (j=0; j<n2; j++) {

This writes out lists of integers.

WSPutInteger32List(stdlink, a[i][j], n3);

}
}

It is important to realize that any expression you create using WSTP functions will be evaluated as soon as it is sent to the Wolfram Language. This means, for example, that if you wanted to transpose an array that you were sending back to the Wolfram Language, all you would need to do is to wrap a (RefLink[Transpose, paclet:ref/Transpose]) around the expression representing the array. You can then do this simply by calling (NonWLLink[WSPutFunction, paclet:ref/c/WSPutFunction])(stdlink,"Transpose",1); just before you start creating the expression that represents the array.

The idea of postprocessing data that you send back to the Wolfram Language has many uses. One example is as a way of sending lists whose length you do not know in advance.

This creates a list in the Wolfram Language by explicitly appending successive elements:

t={};Do[t=Append[t,i^2],{i,5}];t

This creates a list in which each successive element is in a nested sublist:

t={};Do[t={t,i^2},{i,5}];t

(RefLink[Flatten, paclet:ref/Flatten]) flattens out the list:

Flatten[t]

(RefLink[Sequence, paclet:ref/Sequence]) automatically flattens itself:

{Sequence[1,Sequence[4,Sequence[]]]}

In order to call (NonWLLink[WSPutInteger32List(), paclet:ref/c/WSPutInteger32List]), you need to know the length of the list you want to send. But by creating a sequence of nested (RefLink[Sequence, paclet:ref/Sequence]) objects, you can avoid having to know the length of your whole list in advance.

This sets up the (RefLink[List, paclet:ref/List]) around your result.

WSPutFunction(stdlink, "List", 1);

while( condition ) {
     /* generate an element */

Create the next level (RefLink[Sequence, paclet:ref/Sequence]) object.

WSPutFunction(stdlink, "Sequence", 2);

Put the element.

WSPutInteger32(stdlink,  i );

}

This closes off your last (RefLink[Sequence, paclet:ref/Sequence]) object.

WSPutFunction(stdlink, "Sequence", 0);

Basic functions for explicitly getting data from the Wolfram Language.

WSTP provides functions like (NonWLLink[WSPutInteger32(), paclet:ref/c/WSPutInteger32]) to send data from an external program into the Wolfram Language. WSTP also provides functions like (NonWLLink[WSGetInteger32(), paclet:ref/c/WSGetInteger32]) that allow you to get data from the Wolfram Language into an external program.

The list that you give for :ArgumentTypes: in a WSTP template can end with (RefLink[Manual, paclet:ref/Manual]), indicating that after other arguments have been received, you will call WSTP functions to get additional expressions.

:Begin:
:Function:       f

The function f in the Wolfram Language takes 3 arguments.

:Pattern:        f[i_Integer, x_Real, y_Real]

All these arguments are passed directly to the external program.

:Arguments:      {i, x, y}

Only the first argument is sent directly to the external function.

:ArgumentTypes:  {Integer, Manual}

:ReturnType:     Real
:End:

The external function only takes one explicit argument.

double f(int i) {

This declares the variables x and y.

double x, y;

(NonWLLink[WSGetReal64(), paclet:ref/c/WSGetReal64]) explicitly gets data from the link.

WSGetReal64(stdlink, &x);
   WSGetReal64(stdlink, &y);

return i+x+y;
}

WSTP functions such as (NonWLLink[WSGetInteger32, paclet:ref/c/WSGetInteger32])(link,pi) work much like standard C library functions such as fscanf(fp,"%d",pi). The first argument specifies the link from which to get data. The last argument gives the address at which the data that is obtained should be stored.

Getting a function via WSTP.

:Begin:
:Function:       f

The function f in the Wolfram Language takes a list of integers as an argument.

:Pattern:        f[a:{___Integer}]

The list is passed directly to the external program.

:Arguments:      {a}

The argument is to be retrieved manually by the external program.

:ArgumentTypes:  {Manual}

:ReturnType:     Integer
:End:

The external function takes no explicit arguments.

int f(void) {

This declares local variables.

int n, i;
    int a[MAX];

This checks that the function being sent is a list, and stores how many elements it has in n.

WSCheckFunction(stdlink, "List", &n);

This gets each element in the list, storing it in a[i].

for (i=0; i<n; i++)
      WSGetInteger32(stdlink, a+i);

In simple cases, it is usually possible to ensure on the Wolfram Language side that the data you send to an external program has the structure that is expected. But in general the return value from (NonWLLink[WSCheckFunction(), paclet:ref/c/WSCheckFunction]) will be MLSUCCESS only if the data consists of a function with the name you specify.

Note that if you want to get a nested collection of lists or other objects, you can do this by making an appropriate sequence of calls to (NonWLLink[WSCheckFunction(), paclet:ref/c/WSCheckFunction]).

Getting lists of numbers.

When an external program gets data from the Wolfram Language, it must set up a place to store the data. If the data consists of a single integer, as in (NonWLLink[WSGetInteger32, paclet:ref/c/WSGetInteger32])(stdlink,&n), then it suffices just to have declared this integer using intn.

But when the data consists of a list of integers of potentially any length, memory must be allocated to store this list at the time when the external program is actually called.

(NonWLLink[WSGetInteger32List, paclet:ref/c/WSGetInteger32List])(stdlink,&a,&n) will automatically do this allocation, setting a to be a pointer to the result. Note that memory allocated by functions like (NonWLLink[WSGetInteger32List(), paclet:ref/c/WSGetInteger32List]) is always in a special reserved area, so you cannot modify or free it directly.

Here is an external program that will be sent a list of integers.

int f(void) {

This declares local variables. a is an array of integers.

int n;
    int *a;

This gets a list of integers, making a be a pointer to the result.

WSGetInteger32List(stdlink, &a, &n);

This releases the memory used to store the list of integers.

WSReleaseInteger32List(stdlink, a, n);

...
}

If you use IntegerList as an :ArgumentTypes: specification, then WSTP will automatically release the memory used for the list after your external function exits. But if you get a list of integers explicitly using (NonWLLink[WSGetInteger32List(), paclet:ref/c/WSGetInteger32List]), then you must not forget to release the memory used to store the list after you have finished with it.

Getting arrays of numbers.

(NonWLLink[WSGetInteger32List(), paclet:ref/c/WSGetInteger32List]) extracts a one-dimensional array of integers from a single Wolfram Language list. (NonWLLink[WSGetInteger32Array(), paclet:ref/c/WSGetInteger32Array]) extracts an array of integers from a collection of lists or other Wolfram Language functions nested to any depth.

The name of the Wolfram Language function at level i in the structure is stored as a string in heads[i]. The size of the structure at level i is stored in dims[i], while the total depth is stored in d.

If you pass a list of complex numbers to your external program, then (NonWLLink[WSGetReal64Array(), paclet:ref/c/WSGetReal64Array]) will create a two-dimensional array containing a sequence of pairs of real and imaginary parts. In this case, heads[0] will be (StringTypeLink["List", paclet:ref/format/List]) while heads[1] will be "Complex".

Note that you can conveniently exchange arbitrary-precision numbers with external programs by converting them to lists of digits in the Wolfram Language using (RefLink[IntegerDigits, paclet:ref/IntegerDigits]) and (RefLink[RealDigits, paclet:ref/RealDigits]).

Getting character strings and symbol names.

If you use (RefLink[String, paclet:ref/String]) as an :ArgumentTypes: specification, then WSTP will automatically release the memory that is used to store the string after your function exits. This means that if you want to continue to refer to the string, you must allocate memory for it, and explicitly copy each character in it.

If you get a string using (NonWLLink[WSGetString(), paclet:ref/c/WSGetString]), however, then WSTP will not automatically release the memory used for the string when your function exits. As a result, you can continue referring to the string. Be careful not to modify the contents of the string by writing to the memory that is returned by (NonWLLink[WSGetString(), paclet:ref/c/WSGetString]). When you no longer need the string, you must nevertheless explicitly call (NonWLLink[WSReleaseString(), paclet:ref/c/WSReleaseString]) in order to release the memory associated with it.

Getting an arbitrary function.

If you know what function to expect in your external program, then it is usually simpler to call (NonWLLink[WSCheckFunction(), paclet:ref/c/WSCheckFunction]). But if you do not know what function to expect, you have no choice but to call (NonWLLink[WSGetFunction(), paclet:ref/c/WSGetFunction]). If you do this, you need to be sure to call (NonWLLink[WSReleaseSymbol(), paclet:ref/c/WSReleaseSymbol]) to release the memory associated with the name of the function that is found by (NonWLLink[WSGetFunction(), paclet:ref/c/WSGetFunction]).