Resource retrieval
Get the pre-trained net:
NetModel parameters
This model consists of a family of individual nets, each identified by a specific parameter combination. Inspect the available parameters:
Pick a non-default net by specifying the parameters:
Pick a non-default uninitialized net:
Basic usage
Given a piece of text, the RoBERTa net produces a sequence of feature vectors of size 768, which correspond to the sequence of input words or subwords:
Obtain dimensions of the embeddings:
Visualize the embeddings:
Transformer architecture
Each input text segment is first tokenized into words or subwords using a word-piece tokenizer and additional text normalization. Integer codes called token indices are generated from these tokens, together with additional segment indices:
For each input subword token, the encoder yields a pair of indices that correspond to the token index in the vocabulary, and the index of the sentence within the list of input sentences:
The list of tokens always starts with special token index 1, which corresponds to the classification index.
The special token index 3 is used as a separator between the different text segments, marking the end and beginning (except the first) of each sentence. Each subword token is also assigned a positional index:
A lookup is done to map these indices to numeric vectors of size 768:
For each subword token, these three embeddings are combined by summing elements with ThreadingLayer:
The transformer architecture then processes the vectors using 12 structurally identical self-attention blocks stacked in a chain:
The key part of these blocks is the attention module comprising of 12 parallel self-attention transformations, also called “attention heads”:
BERT-like models use self-attention, where the embedding of a given subword depends on the full input text. The following figure compares self-attention (lower left) to other types of connectivity patterns that are popular in deep learning:
Sentence analogies
Define a sentence embedding that takes the last feature vector from RoBERTa subword embeddings (as an arbitrary choice):
Define a list of sentences in two broad categories (food and music):
Precompute the embeddings for a list of sentences:
Visualize the similarity between the sentences using the net as a feature extractor:
Train a classifier model with the subword embeddings
Get a text-processing dataset:
View a random sample of the dataset:
Precompute the RoBERTa vectors for the training and the validation datasets (if available, GPU is highly recommended):
Define a network to classify the sequences of subword embeddings, using a max-pooling strategy:
Train the network on the precomputed vectors from RoBERTa:
Check the classification error rate on the validation data:
Let’s compare the results with the performance of a classifier trained on context-independent word embeddings. Precompute the GloVe vectors for the training and the validation dataset:
Train the classifier on the precomputed GloVe vectors:
Compare the results obtained with RoBERTa and with GloVe:
Net information
Inspect the number of parameters of all arrays in the net:
Obtain the total number of parameters:
Obtain the layer type counts:
Display the summary graphic:
Export to MXNet
Export the net into a format that can be opened in MXNet:
Export also creates a net.params file containing parameters:
Get the size of the parameter file: