Resource retrieval

Get the pre-trained net:

In[1]:=

Out[1]=

Basic usage

Predict the next character in a piece of text:

In[2]:=

Out[2]=

The output values correspond to bytes in the UTF-8 encoding (modulo a subtraction by 1). Decode the prediction:

In[3]:=

Out[3]=

Note that since UTF-8 is a variable-length encoding, decoding single byte values may not always make sense:

In[4]:=

Out[4]=

Multiplicative LSTM and UTF-8 encoding

This model features a non-standard multiplicative LSTM (mLSTM), which can be implemented using NetFoldOperator:

In[5]:=

Out[5]=

Inspect the inner structure of the multiplicative LSTM:

In[6]:=

Out[6]=

This net encodes its input string into a sequence of byte values corresponding to its UTF-8 encoding. Inspect the input encoder:

In[7]:=

Out[7]=

The net predicts the next byte of the sequence. UTF-8 allows single byte values in the range 0-247; hence, there are 248 possible outputs. Inspect the input decoder:

In[8]:=

Out[8]=

Generation

Write a function to generate text efficiently using NetStateObject. Note that FromCharacterCode[…,"UTF-8"] is used at the end, but output byte values are not guaranteed to produce a valid UTF-8 sequence. In this case, FromCharacterCode will issue messages:

In[9]:=

Generate for 100 steps using “This produc” as an initial string:

In[10]:=

Out[10]=

The third optional argument is a “temperature” parameter that scales the input to the final softmax. A high temperature flattens the distribution from which characters are sampled, increasing the probability of extracting less likely characters:

In[11]:=

Out[11]=

Decreasing the temperature sharpens the peaks of the sampling distribution, further decreasing the probability of extracting less likely characters:

In[12]:=

Out[12]=

Very low temperature settings are equivalent to always picking the character with maximum probability. It is typical for sampling to “get stuck in a loop”:

In[13]:=

Out[13]=

Very high temperature settings are equivalent to random sampling. Since the output classes are byte values to be decoded using UTF-8, a very high temperature will almost certainly generate invalid sequences:

In[14]:=

Out[14]=

Sentiment analysis

In this model, a single unit of the output state (unit #2389) turns out to directly reflect the sentiment of the text. The final value of this unit can be used as a feature for sentiment analysis. Define a function to extract it:

In[15]:=

Obtain sentiment scores:

In[16]:=

Out[16]=

In[17]:=

Out[17]=

Get a subset of the movie review dataset:

In[18]:=

In[19]:=

Out[19]=

Obtain the review scores. If available, GPU evaluation (TargetDevice -> “GPU”) is recommended:

In[20]:=

Positive and negative reviews are mostly separated by this single score value:

In[21]:=

Out[21]=

Classify the reviews using the score:

In[22]:=

Out[22]=

Keeping in mind that the training used scalar features from a model not explicitly trained for sentiment analysis, the obtained accuracy is remarkable:

In[23]:=

Out[23]=

Sentiment visualization

Display the evolution of the sentiment cell as the input is read, with red background corresponding to negative sentiment and green to positive:

In[24]:=

As the net reads a positive review, the sentiment cell evolves toward values in the positive score range:

In[25]:=

Out[25]=

In[26]:=

Out[26]=

As the net reads a negative review, the sentiment cell evolves toward values in the negative score range:

In[27]:=

Out[27]=

In[28]:=

Out[28]=

Net information

Inspect the number of parameters of all arrays in the net:

In[29]:=

Out[29]=

Obtain the total number of parameters:

In[30]:=

Out[30]=

Obtain the layer type counts:

In[31]:=

Out[31]=

Export to MXNet

Export the net into a format that can be opened in MXNet:

In[32]:=

Out[32]=

Export also creates a net.params file containing parameters:

In[33]:=

Out[33]=

Get the size of the parameter file:

In[34]:=

Out[34]=

The size is similar to the byte count of the resource object:

In[35]:=

Out[35]=