WXF Format Description

WXF is a binary format for faithfully serializing Wolfram Language expressions in a form suitable for outside storage or interchange with other programs. WXF can readily be interpreted using low-level native types available in many programming languages, making it suitable as a format for reading and writing Wolfram Language expressions in other programming languages.

The basic functions for converting between a Wolfram Language expression and its serialized from are (RefLink[BinarySerialize, paclet:ref/BinarySerialize]) and (RefLink[BinaryDeserialize, paclet:ref/BinaryDeserialize]). Support for reading and writing files with WXF data is built into (RefLink[Export, paclet:ref/Export]) and (RefLink[Import, paclet:ref/Import]).

There are many ways to serialize and deserialize WXF in the Wolfram Language.

Basic Structure

Data in WXF form always contains a plain ASCII header followed by a string of bytes. The header specifies how the bytes can be decoded and is separated by a colon from the string of bytes that represents a sequence of parts.

The byte array continues by giving a sequence of parts, each starting with a token from the following list that specifies the type of the part.

The exhaustive list of WXF tokens.

After the token comes, if necessary, a length specification, followed by the sequence of actual content elements for the part.

Basic Examples

Give the bytes for the serialized form of a symbol:

BinarySerialize[Plot]

Normal[%]

View the bytes as characters in "ISO8859-1":

FromCharacterCode[%]

Give the bytes for the serialized form of a string as characters:

ByteArrayToString[BinarySerialize["hello!"],"ISO8859-1"]

Give the bytes for the serialized form of (RefLink[Range, paclet:ref/Range])[10]:

bytes=Normal[BinarySerialize[Range[10]]]

Create a function to label pieces of the array:

label[data_,label_] :=Labeled[Framed[Row[data,","]],label]

The first two bytes belong to the header, the rest corresponds to the packed array:

Row[{label[bytes[[1;;2]],"header, \":\" "],label[bytes[[3;;]],"packed array"]}]

The header with the version followed by the separator:

FromCharacterCode[{56,58}]

The packed array with the type, the rank and dimensions, and finally the data:

Row[{label[bytes[[3;;4]],"type of array"],label[bytes[[5;;6]],"  rank, dimension  "],label[bytes[[7;;]],"data"]}]

Examples with Multiple Parts

The examples in the previous section essentially consisted of a single part. There was a single token from the token list, followed by size information, followed by the data. The examples in this section contain multiple parts and types. The first example is shown by the following interactive illustration:

bytes=BinarySerialize[Select[OddQ]]

The first two bytes are the header and colon separator:

{FromCharacterCode[bytes[[1]]],FromCharacterCode[bytes[[2]]]}

The next byte is the token for a function, followed by the number of arguments:

{FromCharacterCode[bytes[[3]]],bytes[[4]]}

The next byte is the token of the function's head, in this case symbol:

FromCharacterCode[bytes[[5]]]

After that is the number of bytes in the UTF-8 representation of the symbol:

bytes[[6]]

Therefore, the next 6 bytes are the UTF-8 representation of the symbol:

ByteArrayToString[bytes[[7;;12]]]

The complete head being read, the next byte is the token of the first argument:

FromCharacterCode[bytes[[13]]]

As this argument is a symbol, the next byte will be its length, followed by its name:

{bytes[[14]],ByteArrayToString[bytes[[15;;18]]]}

The function only had one argument, so all the bytes must be read by now:

Length[bytes]==18

The second example is a list of three elements. The list is represented as a function of length 3, with head (RefLink[List, paclet:ref/List]) followed by three parts. The first parts introduce the format used to represent integers in a compact form using a token and an (StringTypeLink["Integer8", paclet:ref/format/Integer8]). The last part shows the representation of a (RefLink[ByteArray, paclet:ref/ByteArray]):

Give the bytes for the serialized form of {1,-1,(RefLink[ByteArray, paclet:ref/ByteArray])[{1,2,3}]}:

list=Normal[BinarySerialize[{1,-1,ByteArray[{1,2,3}]}]]

Ignoring the header, this expression is a function of three arguments:

{FromCharacterCode[list[[3]]] , list[[4]]}

The head is a symbol of four bytes, namely (RefLink[List, paclet:ref/List]):

Row[{label[list[[{5}]],"s"] , label[list[[{6}]],"length"],label[list[[7;;10]],"List"]},Spacer[.5]]

After the head comes the first argument, namely the 8-bit integer 1:

{FromCharacterCode[list[[11]]] , list[[12]]}

The next two bytes are the second argument, namely the 8-bit integer -1:

{FromCharacterCode[list[[13]]] , list[[14]]}

Use (RefLink[Mod, paclet:ref/Mod]) to interpret the byte as a signed value, converting 255 to the expected -1:

Mod[list[[14]],256,-128]

The third and final argument is a binary string of length 3 with values {1,2,3}:

Row[{label[list[[{15}]],"B"] , label[list[[{16}]],"length"],label[list[[17;;19]],"values"]},Spacer[.5]]

More on the Header

The header is a plain ASCII string of variable length, delimited by the character ":". For the current version (1.0) of WXF, the first byte in the header is the character "8" (i.e. byte value 56). When the binary serialization is zip compressed, this is indicated in the header by the character (StringTypeLink["C", paclet:ref/format/C]). The header is never compressed; the compression only applies to the following string of bytes.

Serialize an expression to give a byte array:

BinarySerialize["body"]

The first byte is the character 8:

ByteArrayToString[%,"ISO8859-1"]

Compression is indicated by the second character of the header being "C":

ByteArrayToString[BinarySerialize["body",PerformanceGoal->"Size"],"ISO8859-1"]

Length Encoding (Varint)

WXF types fall into three categories:

Atomic types of fixed length like machine-precision (RefLink[Integer, paclet:ref/Integer]) or (RefLink[Real, paclet:ref/Real]).

Atomic types of variable length like (RefLink[String, paclet:ref/String]) or (RefLink[Symbol, paclet:ref/Symbol]).

Types with a variable number of subparts like general expressions (token type "f" for function) or (RefLink[Association, paclet:ref/Association]).

In WXF, all integers representing a length or a size are serialized using the varint method. A varint is a self-indicating variable-length format, where smaller integers require fewer bytes. Each byte except the last one has its most significant bit (MSB) set. The MSB indicates if the following byte of the stream is also part of the varint, acting as a continuation marker. The lower seven bits of each byte store the binary representation of the integer, with the least significant group first.

Here is an example of constructing the varint representation of 500:

Base-2 digits of 500:

bits=IntegerDigits[500, 2]

Group bits by 7, with the least significant bits first:

grouped7=Partition[Reverse[bits], UpTo[7]]

Order bits of each group, with the most significant bit first, and pad each group so that they have 8 bits each:

grouped8=Map[Composition[PadLeft[#,8]&,Reverse],grouped7]

Set the first bit of each byte except the last one to get the varint binary form:

varint=ReplacePart[grouped8, {i_,1} /; i<Length[grouped8]:>1]

Construct byte values from base-2 digits, to give the varint representation of 500:

Map[FromDigits[#,2]&, varint]

The reverse operation of decoding the varint encoded byte sequence {244,3} back to 500 is explained in the following illustration, in which each initial varint bit is assigned a unique color:

Strings, Symbols and Non-Machine Numbers

Strings, symbols and non-machine numbers are represented using the same format. The first byte is a token, then the byte count encoded in the varint format, followed by a string of Unicode characters encoded as UTF-8 corresponding to the string (RefLink[InputForm, paclet:ref/InputForm]) of the expression. A non-machine number can be either an arbitrary-precision real or an integer that requires more than (RefLink[$SystemWordLength, paclet:ref/$SystemWordLength]) bits to represent.

Types based on (RefLink[InputForm, paclet:ref/InputForm]).

Serialize the first 500 characters of Alice in Wonderland:

bytes=BinarySerialize[StringTake[ExampleData[{"Text","AliceInWonderland"}],500]]

The first byte after the header is the token for a string:

FromCharacterCode[bytes[[3]]]

The next two bytes are 500 in the varint encoding as seen in the preceding example:

Normal[bytes[[4;;5]]]

The remaining bytes are the string contents in UTF-8:

ByteArrayToString[bytes[[6;;-1]]]

Serialize a non-machine integer:

bytes=BinarySerialize[10^127]

Other than the token, the serialization is the same as for a string:

{FromCharacterCode[bytes[[3]]],Normal[bytes[[4;;5]]],ByteArrayToString[bytes[[6;;-1]]]}

A number with one fewer digit requires only a single byte to encode the length:

bytes=BinarySerialize[10^126];

{FromCharacterCode[bytes[[3]]],bytes[[4]],ByteArrayToString[bytes[[5;;-1]]]}

Machine Integers Serialization

Machine integers are identified by the smallest integer type from the following list that can represent the value, followed by the two's complement representation of the integer. The byte ordering is always little endian.

Integer tokens, their associated types and the number of bytes used by each representation.

The bytes corresponding to the serialization of 2^14:

bin=BinarySerialize[2^14]

Skip the two-byte header and display the first token as a character:

FromCharacterCode[bin[[3]]]

View the bytes after the token:

Normal[bin[[4;;-1]]]

Convert the preceding pair of bytes to an integer:

bin[[4]] + BitShiftLeft[bin[[5]],8] == 2^14

Negative integers binary representation uses the two's complement method. Given a N-bit integer α, its two's complement β is its complement with respect to (2^(RefLink[N, paclet:ref/N])): α+β=(2^(RefLink[N, paclet:ref/N])). Negation of a number is performed by taking the two's complement.

The two's complement of the 8-bit integer 1:

2^8-1

The two's complement is the 8-bit binary representation of -1:

Normal[ExportByteArray[-1, "Integer8"]]

Serialize a negative 16-bit integer:

bytes=BinarySerialize[-10000]

The last two bytes are the integer value:

bytes[[-2;;]]//Normal

Convert the pair of bytes to its decimal form:

bytes[[-2]] + BitShiftLeft[bytes[[-1]], 8]

The value is the 16-bit two's complement of 10000:

10000+% == 2^16

Machine Reals Serialization

Machine reals are represented using the character "r" followed by the memory representation of a double floating-point value in the IEEE 754 standard.

Serialize a real number:

bytes=BinarySerialize[4.]

Skip the two-byte header and display the first token as a byte and as a character:

bytes[[3]]

FromCharacterCode[bytes[[3]]]

The next bytes are the real value in the IEEE 754 standard:

Normal[bytes[[4;;]]]

ExportByteArray[4.,"Real64"] == bytes[[4;;]]

Machine-precision complex numbers are serialized as a function of two machine-precision reals. The following illustration highlights the (RefLink[Complex, paclet:ref/Complex]) head followed by two real values:

Serialize a machine complex number:

bytes=Normal[BinarySerialize[4. + 4.I]]

The first bytes after the header are a function of length 2 with head (RefLink[Complex, paclet:ref/Complex]):

Row[{label[{bytes[[3]]}, "f"], label[{bytes[[4]]}," length "],label[bytes[[5;;13]], "Complex"]}]

The next nine bytes are the real part and match the serialization of the real value 4. as shown in the previous example:

bytes[[14;;22]]

The remaining bytes are the imaginary part, again the real value 4.:

bytes[[23;;]]

Function Serialization

Functions are represented in WXF by the character "f", followed by the expression length in the varint format. The number of elements is equal to the length incremented by one, for the head. The head and the parts are arbitrary serialized expressions. In particular, the head can also be a function: (RefLink[Select, paclet:ref/Select])[(RefLink[OddQ, paclet:ref/OddQ])][{1,2,3}] is a function of length 1 with head (RefLink[Select, paclet:ref/Select])[(RefLink[OddQ, paclet:ref/OddQ])], which itself is a function with head (RefLink[Select, paclet:ref/Select]) and length 1.

Serialize an expression, using (RefLink[Unevaluated, paclet:ref/Unevaluated]) to prevent it from evaluating:

bytes=BinarySerialize[Unevaluated@Select[OddQ][{1,2,3}]]

The first two bytes correspond to a function of length 1:

{FromCharacterCode[bytes[[3]]],bytes[[4]]}

The next 16 bytes are the serialization of the head (RefLink[Select, paclet:ref/Select])[(RefLink[OddQ, paclet:ref/OddQ])] shown previously:

Normal[bytes[[5;;20]]]

The remaining bytes are the argument, with the token and length first:

{FromCharacterCode[bytes[[21]]],bytes[[22]]}

Followed by the head:

{FromCharacterCode[bytes[[23]]],bytes[[24]],ByteArrayToString[bytes[[25;;28]]]}

The three machine integers round out the expression:

Normal[bytes[[29;;-1]]]

Associations Serialization

Associations are represented by the character "A", followed by the length and the rules.

An association's rules are represented by the character "-", and delayed rules by the character ":". It is immediately followed by two arbitrary serialized expressions. The length of the association's rule is always two and thus is omitted. The following illustration shows the serialization of a simple association:

Serialize the association:

bytes=Normal[BinarySerialize[<|1->2, "a":>"b"|>]]

The first bytes correspond to an association of two elements:

{FromCharacterCode[bytes[[3]]],bytes[[4]]}

Then comes the rule that implicitly has two parts; the length is thus omitted:

Row[{label[{FromCharacterCode[bytes[[5]]]},"Rule"],label[bytes[[6;;7]],"Key"],label[bytes[[8;;9]],"Value"]},Spacer[.5]]

The delayed rule is the final part:

Row[{label[{FromCharacterCode[bytes[[10]]]},"RuleDelayed"],label[bytes[[11;;13]],"Key"],label[bytes[[14;;16]],"Value"]},Spacer[.5]]

Rules of the previous example were part of an association. (RefLink[Rule, paclet:ref/Rule]) and (RefLink[RuleDelayed, paclet:ref/RuleDelayed]) that are not part of an (RefLink[Association, paclet:ref/Association]) are serialized as functions. The serialized form is less packed, as shown in the next example.

Serialize a list of rules:

bytes=Normal[BinarySerialize[{1->2, "a":>"b"}]]

The first bytes declare a list of two elements:

Row[{label[{FromCharacterCode[bytes[[3]]]},"f"],label[{bytes[[4]]},"length"],label[bytes[[5;;10]],"List"]},Spacer[.5]]

The first element is a function of length 2 with head (RefLink[Rule, paclet:ref/Rule]):

Row[{label[{FromCharacterCode[bytes[[11]]]},"f"],label[{bytes[[12]]}," length "],label[{FromCharacterCode[bytes[[13]]]}," Symbol "],label[{bytes[[14]]}," length "],label[bytes[[15;;18]],"Rule"]}]

The rule's arguments remain the same as in the association case:

bytes[[19;;22]]

The second element is also a function of length 2, but its head is (RefLink[RuleDelayed, paclet:ref/RuleDelayed]):

Row[{label[{FromCharacterCode[bytes[[23]]]},"function"],label[{bytes[[24]]}," length "],label[{FromCharacterCode[bytes[[25]]]}," Symbol "],label[{bytes[[26]]}," length "],label[bytes[[27;;37]],"RuleDelayed"]}]

Similarly, the elements of the rule delayed remain unchanged:

bytes[[38;;-1]]

The serialized length is roughly the size of the string (RefLink[FullForm, paclet:ref/FullForm]):

ToString[FullForm[{1->2,"a":>"b"}]]//StringLength

The association serializes to a more compact form:

BinarySerialize[<|1->2, "a":>"b"|>]//Length

Binary Strings

Binary strings are represented by the token "B". They follow the same pattern as strings, but the byte sequence is arbitrary rather than UTF-8 characters. A (RefLink[ByteArray, paclet:ref/ByteArray]) is serialized as a binary string.

Serialize a byte array:

bytes=Normal[BinarySerialize[ByteArray[{0,42,192}]]]

The first byte after the header is a binary string token, followed by the length of the binary data:

Row[{label[{bytes[[3]]}, "B"], label[{bytes[[4]]}, "length"]},Spacer[.1]]

The next bytes are the data:

bytes[[5;;]]

Decoding the bytes as UTF-8 does not always succeed:

FromCharacterCode[bytes[[5;;]], "UTF-8"];

The bytes can always be represented as a string using the "ISO8859-1" encoding:

FromCharacterCode[bytes[[5;;]], "ISO8859-1"]

Numeric Arrays

Arrays are multidimensional tables of machine-precision numeric values. Arrays are represented by the following sequence: a token specifying the type of array, a token specifying the type of the values, the rank in the varint format, the dimensions as a sequence of integers also in the varint format and finally, the data.

There are two types of arrays in the WXF format: packed arrays represented by the token "Á" (byte value 193) and numeric arrays represented by the token "Â" (byte value 194). There are slight differences between the two, the major one being the supported value type, as described in the following tables.

The valid value type tokens for packed arrays.

The valid value type tokens for numeric arrays.

The integer range supported by packed arrays varies with the system word length, (RefLink[$SystemWordLength, paclet:ref/$SystemWordLength]), from -(2^31)+1 to (2^31)-1 on a 32-bit environment, and from -(2^63)+1 to (2^63)-1 on a 64-bit environment. Packed arrays of reals cannot store IEEE exceptions NaN and inf. These restrictions do not apply to numeric arrays.

Define a matrix:

matrix = Table[1000i+j,{i,2},{j,3}]

Serialize it as a packed array:

bytes=Normal[BinarySerialize[Developer`ToPackedArray[matrix]]]

The first byte after the header is the packed array token:

bytes[[3]]

The next byte indicates an array of 16-bit signed integers:

bytes[[4]]

The following bytes are the rank and dimensions of the array:

Row[{label[{bytes[[5]]}, "rank"], label[bytes[[6;;7]], "dimensions"]},Spacer[.1]]

The trailing bytes are the values:

bytes[[8;;]]

It is possible to reconstruct the decimal form of each 16-bit integer. First, group the pair of bytes:

int16=Partition[bytes[[8;;]],2]

Each pair is a little endian 16-bit long integer whose value is reconstructed using a bit shift operation:

Map[First[#] + BitShiftLeft[Last[#],8]&,int16]

The interactive illustration following shows the serialization of the previous matrix before packing. The sequence of elements is significantly different, since it involves nested functions with head (RefLink[List, paclet:ref/List]). The inner lists have three parts corresponding to the integer values. It is worth noting that the binary representation of the integer values is similar to the one witnessed in the packed array case (little endian signed 16-bit integer).

Array value type tokens are constructed as bit fields. The four least significant bits store the log of the size of the numeric type in bytes, and the four most significant bits represent the numeric type.

The four least significant bits of the value type token and the corresponding type sizes in bytes.

The four most significant bits of the value type token and the corresponding numeric types.

It is possible to construct the bit field corresponding to an array of double-precision reals using the Wolfram Language.

A double-precision real is 8 bytes long; the logarithm to base 2 is 3:

Log[2,8]

Find the bit representation:

lsb=IntegerDigits[%, 2, 4]

View the bit field made from the concatenation of the numeric type corresponding to a real number and the type size:

msb={0,0,1,0};

Row[{label[msb,"real type"], label[lsb,"  8 byte long  "]}]

Convert the preceding bit sequence to retrieve the expected byte value:

FromDigits[Join[msb, lsb],2]==35