Wolfram Research

Function Repository Resource:

BinarySerializeWithDefinitions

Source Notebook

Serialize an expression along with any dependent definitions

Contributed by: Richard Hennigan (Wolfram Research)

ResourceFunction["BinarySerializeWithDefinitions"][expr]

serializes expr to a ByteArray along with its dependent definitions.

Details and Options

ResourceFunction["BinarySerializeWithDefinitions"] has the following options:
Method Automatic details of serialization methods to use
PerformanceGoal Automatic aspects of performance to try to optimize
Possible settings for PerformanceGoal include:
"Speed" optimize for serialization and deserialization speed
"Size" optimize for smallness of serialized output
Automatic automatically pick serialization strategy
Possible settings for Method include:
{ typespec 1 enc 1 , } specify encodings for particular types
Automatic pick encodings automatically based on data
Possible forms for the typespeci include:
"PackedArrayIntegerType" integer packed arrays
"PackedArrayRealType" real-valued packed arrays
"PackedArrayComplexType" complex-valued packed arrays
Possible settings for "PackedArrayIntegerType" include "Integer8", "Integer16", "Integer32" and "Integer64" (64-bit systems only).
Possible settings for "PackedArrayRealType" include "Real32" and "Real64".
Possible settings for "PackedArrayComplexType" include "Complex64" and "Complex128".
The default for the enci encoding specifications is Automatic, in which case ResourceFunction["BinarySerializeWithDefinitions"] will use a type that fits the data it is given.
ResourceFunction["BinarySerializeWithDefinitions"] uses the WXF format.

Examples

Basic Examples

Serialize an expression with dependent definitions:

In[1]:=
f[x_] := g[x] + 1;
g[x_] := 2 x;
bytes = ResourceFunction["BinarySerializeWithDefinitions"][f]
Out[3]=

Definitions are restored when deserialized:

In[4]:=
ClearAll[f, g];
BinaryDeserialize[bytes][5]
Out[5]=

Generalizations and Extensions

The serialized expression is an evaluation that restores definitions before returning the output:

In[6]:=
ClearAll[f, g];
f[x_] := g[x] + 1;
g[x_] := 2 x;
bytes = ResourceFunction["BinarySerializeWithDefinitions"][f];
In[7]:=
BinaryDeserialize[bytes, HoldForm]
Out[7]=

BinarySerialize by itself does not preserve definitions:

In[8]:=
b1 = BinarySerialize[Unevaluated[f[5]]]
b2 = ResourceFunction["BinarySerializeWithDefinitions"][
  Unevaluated[f[5]]]
Out[8]=
Out[9]=
In[10]:=
ClearAll[f, g];
In[11]:=
BinaryDeserialize[b1]
Out[11]=
In[12]:=
BinaryDeserialize[b2]
Out[12]=

Options

PerformanceGoal

Serialize a Dataset:

In[13]:=
bin = ResourceFunction["BinarySerializeWithDefinitions"][
  ResourceData["Meteorite Landings"]]
Out[13]=

Serialize the same Dataset with PerformanceGoal set to "Size":

In[14]:=
compressed = ResourceFunction["BinarySerializeWithDefinitions"][
  ResourceData["Meteorite Landings"], PerformanceGoal -> "Size"]
Out[14]=

Both forms represent the same expression:

In[15]:=
BinaryDeserialize[bin] == BinaryDeserialize[compressed]
Out[15]=

Method

Create a packed array of integer values:

In[16]:=
integers = ConstantArray[2^14, 3];

By default, BinarySerializeWithDefinitions uses the smallest integer type that fits the data:

In[17]:=
ResourceFunction["BinarySerializeWithDefinitions"][integers]
Out[17]=

Serialize the packed array using a bigger integer type:

In[18]:=
ResourceFunction["BinarySerializeWithDefinitions"][integers, Method -> "PackedArrayIntegerType" -> "Integer32"]
Out[18]=

Create a packed array of real values:

In[19]:=
reals = RandomReal[1, 3]
Out[19]=

Serialize the array:

In[20]:=
real64 = ResourceFunction["BinarySerializeWithDefinitions"][reals]
Out[20]=

Serialize the array using machine floats, trading precision for a smaller output:

In[21]:=
real32 = ResourceFunction["BinarySerializeWithDefinitions"][reals, Method -> "PackedArrayRealType" -> "Real32"]
Out[21]=

Create a packed array of complex values:

In[22]:=
complexes = RandomComplex[1 + I, 3]
Out[22]=

Serialize the array:

In[23]:=
ResourceFunction["BinarySerializeWithDefinitions"][complexes]
Out[23]=

Serialize the array using lower precision:

In[24]:=
ResourceFunction["BinarySerializeWithDefinitions"][complexes, Method -> {"PackedArrayComplexType" -> "Complex64"}]
Out[24]=

Applications

CloudPut an evaluation that runs each time you use CloudGet on it:

In[25]:=
CloudSymbol["count"] = 0;
info[] := With[{i = <|"Time" -> Now, "Count" -> (CloudSymbol["count"] += 1)|>}, PutAppend[i, CloudObject["log.wl"]]; i];
In[26]:=
With[{b = ResourceFunction["BinarySerializeWithDefinitions"][
    Unevaluated[info[]]]},
 CloudPut[Unevaluated[BinaryDeserialize[b]], "info"]
 ]
Out[26]=
In[27]:=
CloudGet["info"]
Out[27]=
In[28]:=
CloudGet["info"]
Out[28]=

Check the log:

In[29]:=
ReadList[CloudObject["log.wl"]]
Out[29]=

Properties and Relations

When there are no dependent symbols, BinarySerializeWithDefinitions will produce the same ByteArray as BinarySerialize:

In[30]:=
ints = RandomInteger[1000, 100];
ResourceFunction["BinarySerializeWithDefinitions"][ints] === BinarySerialize[ints]
Out[31]=

The results are different when definitions need to be included:

In[32]:=
f[x_] := x + 1;
ResourceFunction["BinarySerializeWithDefinitions"][
  Unevaluated[f[ints]]] === BinarySerialize[Unevaluated[f[ints]]]
Out[33]=

Possible Issues

For small expressions, definitions add a significant amount of overhead relative to the total size:

In[34]:=
x = 1;
ResourceFunction["BinarySerializeWithDefinitions"][x]
ResourceFunction["BinarySerializeWithDefinitions"][Unevaluated[x]]
Out[35]=
Out[36]=

This overhead becomes insignificant for larger expressions:

In[37]:=
y = Range[100000];
ResourceFunction["BinarySerializeWithDefinitions"][y]
ResourceFunction["BinarySerializeWithDefinitions"][Unevaluated[y]]
Out[38]=
Out[39]=

Requirements

Wolfram Language 11.3 (March 2018) or above

Resource History

See Also

License Information