Wolfram Function Repository

Instant-use add-on functions for the Wolfram Language

Function Repository Resource:

 BinarySerializeWithDefinitions

Source Notebook

Serialize an expression along with any dependent definitions

Contributed by: Richard Hennigan (Wolfram Research)

ResourceFunction["BinarySerializeWithDefinitions"][expr]

serializes expr to a ByteArray object along with its dependent definitions.

Details and Options

ResourceFunction["BinarySerializeWithDefinitions"] has the following options:
MethodAutomaticdetails of serialization methods to use
PerformanceGoalAutomaticaspects of performance to try to optimize
Possible settings for PerformanceGoal include:
"Speed"optimize for serialization and deserialization speed
"Size"optimize for smallness of serialized output
Automaticautomatically pick serialization strategy
Possible settings for Method include:
{typespec1→enc1,…}specify encodings for particular types
Automaticpick encodings automatically based on data
Possible forms for the typespeci include:
"PackedArrayIntegerType"integer packed arrays
"PackedArrayRealType"real-valued packed arrays
"PackedArrayComplexType"complex-valued packed arrays
Possible settings for "PackedArrayIntegerType" include "Integer8", "Integer16", "Integer32" and "Integer64" (64-bit systems only).
Possible settings for "PackedArrayRealType" include "Real32" and "Real64".
Possible settings for "PackedArrayComplexType" include "Complex64" and "Complex128".
The default for the enci encoding specifications is Automatic, in which case ResourceFunction["BinarySerializeWithDefinitions"] will use a type that fits the data it is given.
ResourceFunction["BinarySerializeWithDefinitions"] uses the WXF format.

Examples

Basic Examples (2) 

Serialize an expression with dependent definitions:

In[1]:=
f[x_] := g[x] + 1; g[x_] := 2 x; bytes = ResourceFunction["BinarySerializeWithDefinitions"][f]
Out[3]=

Definitions are restored when deserialized:

In[4]:=
ClearAll[f, g]; BinaryDeserialize[bytes][5]
Out[5]=

Generalizations and Extensions (2) 

The serialized expression is an evaluation that restores definitions before returning the output:

In[6]:=
ClearAll[f, g]; f[x_] := g[x] + 1; g[x_] := 2 x; bytes = ResourceFunction["BinarySerializeWithDefinitions"][f];
In[7]:=
BinaryDeserialize[bytes, HoldForm]
Out[7]=

BinarySerialize by itself does not preserve definitions:

In[8]:=
b1 = BinarySerialize[Unevaluated[f[5]]]
Out[8]=
In[9]:=
b2 = ResourceFunction["BinarySerializeWithDefinitions"][ Unevaluated[f[5]]]
Out[9]=
In[10]:=
ClearAll[f, g];
In[11]:=
BinaryDeserialize[b1]
Out[11]=
In[12]:=
BinaryDeserialize[b2]
Out[12]=

Options (6) 

PerformanceGoal (3) 

Serialize a Dataset:

In[13]:=
bin = ResourceFunction["BinarySerializeWithDefinitions"][ ResourceData["Meteorite Landings"]]
Out[13]=

Serialize the same Dataset with PerformanceGoal set to "Size":

In[14]:=
compressed = ResourceFunction["BinarySerializeWithDefinitions"][ ResourceData["Meteorite Landings"], PerformanceGoal -> "Size"]
Out[14]=

Both forms represent the same expression:

In[15]:=
BinaryDeserialize[bin] == BinaryDeserialize[compressed]
Out[15]=

Method (3) 

Create a packed array of integer values:

In[16]:=
integers = ConstantArray[2^14, 3];

By default, BinarySerializeWithDefinitions uses the smallest integer type that fits the data:

In[17]:=
ResourceFunction["BinarySerializeWithDefinitions"][integers]
Out[17]=

Serialize the packed array using a bigger integer type:

In[18]:=
ResourceFunction["BinarySerializeWithDefinitions"][integers, Method -> "PackedArrayIntegerType" -> "Integer32"]
Out[18]=

Create a packed array of real values:

In[19]:=
reals = RandomReal[1, 3]
Out[19]=

Serialize the array:

In[20]:=
real64 = ResourceFunction["BinarySerializeWithDefinitions"][reals]
Out[20]=

Serialize the array using machine floats, trading precision for a smaller output:

In[21]:=
real32 = ResourceFunction["BinarySerializeWithDefinitions"][reals, Method -> "PackedArrayRealType" -> "Real32"]
Out[21]=

Create a packed array of complex values:

In[22]:=
complexes = RandomComplex[1 + I, 3]
Out[22]=

Serialize the array:

In[23]:=
ResourceFunction["BinarySerializeWithDefinitions"][complexes]
Out[23]=

Serialize the array using lower precision:

In[24]:=
ResourceFunction["BinarySerializeWithDefinitions"][complexes, Method -> {"PackedArrayComplexType" -> "Complex64"}]
Out[24]=

Applications (2) 

CloudPut an evaluation that runs each time you use CloudGet on it:

In[25]:=
CloudSymbol["count"] = 0; info[] := With[{i = <|"Time" -> Now, "Count" -> (CloudSymbol["count"] += 1)|>}, PutAppend[i, CloudObject["log.wl"]]; i];
In[26]:=
With[{b = ResourceFunction["BinarySerializeWithDefinitions"][ Unevaluated[info[]]]}, CloudPut[Unevaluated[BinaryDeserialize[b]], "info"] ]
Out[26]=
In[27]:=
CloudGet["info"]
Out[27]=
In[28]:=
CloudGet["info"]
Out[28]=

Check the log:

In[29]:=
ReadList[CloudObject["log.wl"]]
Out[29]=

Properties and Relations (2) 

When there are no dependent symbols, BinarySerializeWithDefinitions will produce the same ByteArray as BinarySerialize:

In[30]:=
ints = RandomInteger[1000, 100]; ResourceFunction["BinarySerializeWithDefinitions"][ints] === BinarySerialize[ints]
Out[23]=

The results are different when definitions need to be included:

In[31]:=
f[x_] := x + 1; ResourceFunction["BinarySerializeWithDefinitions"][ Unevaluated[f[ints]]] === BinarySerialize[Unevaluated[f[ints]]]
Out[27]=

Possible Issues (2) 

For small expressions, definitions add a significant amount of overhead relative to the total size:

In[32]:=
x = 1;
In[33]:=
ResourceFunction["BinarySerializeWithDefinitions"][x]
Out[33]=
In[34]:=
ResourceFunction["BinarySerializeWithDefinitions"][Unevaluated[x]]
Out[34]=

This overhead becomes insignificant for larger expressions:

In[35]:=
y = Range[100000];
In[36]:=
ResourceFunction["BinarySerializeWithDefinitions"][y]
Out[36]=
In[37]:=
ResourceFunction["BinarySerializeWithDefinitions"][Unevaluated[y]]
Out[37]=

Requirements

Wolfram Language 11.3 (March 2018) or above

Version History

  • 2.0.0 – 26 December 2019
  • 1.0.0 – 12 October 2018

Related Resources

License Information