Wolfram Neural Net Repository

Immediate Computable Access to Neural Net Models

ConceptNet Numberbatch Word Vectors V17.06 (Raw Model)

Represent words as vectors

Released in 2016, these word representations were obtained by combining knowledge from the human-made ConceptNet graph and multiple pre-trained, distributional-based embeddings: GloVe, word2vec and the fastText algorithm trained on the Open Subtitles 2016 dataset. This net encodes more than 400,000 tokens as unique vectors, with all tokens outside the vocabulary encoded as the zero-vector.

Number of layers: 1 | Parameter count: 125,158,200 | Trained size: 503 MB |

Training Set Information

Examples

Resource retrieval

Get the pre-trained net:

In[1]:=
NetModel["ConceptNet Numberbatch Word Vectors V17.06 (Raw Model)"]
Out[1]=

Basic usage

Use the net to obtain a list of word vectors:

In[2]:=
vectors = NetModel[ "ConceptNet Numberbatch Word Vectors V17.06 (Raw Model)"][{"hello", "world"}]
Out[2]=

Obtain the dimensions of the vectors:

In[3]:=
Dimensions[vectors]
Out[3]=

Use the embedding layer inside a NetChain:

In[4]:=
chain = NetChain[{NetMapOperator@ NetModel[ "ConceptNet Numberbatch Word Vectors V17.06 (Raw Model)"], LongShortTermMemoryLayer[10]}]
Out[4]=

Feature visualization

Create two lists of related words:

In[5]:=
animals = {"alligator", "ant", "bear", "bee", "bird", "camel", "cat", "cheetah", "chicken", "chimpanzee", "cow", "crocodile", "deer", "dog", "dolphin", "duck", "eagle", "elephant", "fish", "fly"};
In[6]:=
fruits = {"apple", "apricot", "avocado", "banana", "blackberry", "blueberry", "cherry", "coconut", "cranberry", "grape", "turnip", "mango", "melon", "papaya", "peach", "pineapple", "raspberry", "strawberry", "ribes", "fig"};

Visualize relationships between the words using the net as a feature extractor:

In[7]:=
FeatureSpacePlot[Join[animals, fruits], FeatureExtractor -> NetModel["ConceptNet Numberbatch Word Vectors V17.06 (Raw Model)"]]
Out[7]=

Word analogies

Get the pre-trained net:

In[8]:=
net = NetModel[ "ConceptNet Numberbatch Word Vectors V17.06 (Raw Model)"]
Out[8]=

Get a list of tokens:

In[9]:=
words = NetExtract[net, "Input"][["Labels"]]
Out[9]=

Obtain the vectors:

In[10]:=
vecs = Normal@NetExtract[net, "Weights"];

Create an association whose keys are tokens and whose values are vectors:

In[11]:=
word2vec = AssociationThread[words -> vecs];

Find the eight nearest tokens to "king":

In[12]:=
Nearest[word2vec, word2vec["king"], 8]
Out[12]=

France is to Paris as Germany is to:

In[13]:=
Nearest[word2vec, word2vec["paris"] - word2vec["france"] + word2vec["germany"], 5]
Out[13]=

Net information

Inspect the number of parameters of all arrays in the net:

In[14]:=
NetInformation[ NetModel[ "ConceptNet Numberbatch Word Vectors V17.06 (Raw Model)"], "ArraysElementCounts"]
Out[14]=

Obtain the total number of parameters:

In[15]:=
NetInformation[ NetModel[ "ConceptNet Numberbatch Word Vectors V17.06 (Raw Model)"], "ArraysTotalElementCount"]
Out[15]=

Obtain the layer type counts:

In[16]:=
NetInformation[ NetModel[ "ConceptNet Numberbatch Word Vectors V17.06 (Raw Model)"], "LayerTypeCounts"]
Out[16]=

Export to MXNet

Export the net into a format that can be opened in MXNet:

In[17]:=
jsonPath = Export[FileNameJoin[{$TemporaryDirectory, "net.json"}], NetModel["ConceptNet Numberbatch Word Vectors V17.06 (Raw Model)"], "MXNet"]
Out[17]=

Export also creates a net.params file containing parameters:

In[18]:=
paramPath = FileNameJoin[{DirectoryName[jsonPath], "net.params"}]
Out[18]=

Get the size of the parameter file:

In[19]:=
FileByteCount[paramPath]
Out[19]=

The size is similar to the byte count of the resource object:

In[20]:=
ResourceObject[ "ConceptNet Numberbatch Word Vectors V17.06 (Raw Model)"]["ByteCount"]
Out[20]=

Represent the MXNet net as a graph:

In[21]:=
Import[jsonPath, {"MXNet", "NodeGraphPlot"}]
Out[21]=

Requirements

Wolfram Language 11.2 (September 2017) or above

Resource History

  • Date Created: 8 September 2017
  • Latest Update: 24 September 2018

Reference