Wolfram Computation Meets Knowledge

ResNet-101 Trained on ImageNet Competition Data

Identify the main object in an image

Released in 2015 by Microsoft Research Asia, the ResNet architecture (with its three realizations ResNet-50, ResNet-101 and ResNet-152) obtained very successful results in the ImageNet and MS-COCO competition. The core idea exploited in these models, residual connections, is found to greatly improve gradient flow, thus allowing training of much deeper models, with tens or even hundreds of layers. ImageNet classes are mapped to Wolfram Language Entities through their unique WordNet IDs.

Number of layers: 347 | Parameter count: 44,654,504 | Trained size: 180 MB

Training Set Information

Performance

Examples

Resource retrieval

Retrieve the resource object:

In[1]:=
ResourceObject["ResNet-101 Trained on ImageNet Competition Data"]
Out[1]=

Get the pre-trained net:

In[2]:=
NetModel["ResNet-101 Trained on ImageNet Competition Data"]
Out[2]=

Basic usage

Classify an image:

In[3]:=
CloudGet["https://www.wolframcloud.com/objects/e26e0c8b-d8c3-4011-805e-848cd307ac54"] (* Evaluate this cell to copy the example input from a cloud object *)
Out[3]=

The prediction is an Entity object, which can be queried:

In[4]:=
pred["Definition"]
Out[4]=

Get a list of available properties of the predicted Entity:

In[5]:=
pred["Properties"]
Out[5]=

Obtain the probabilities of the ten most likely entities predicted by the net:

In[6]:=
CloudGet["https://www.wolframcloud.com/objects/5f576d72-b7cf-4606-bacf-e8da786cd513"] (* Evaluate this cell to copy the example input from a cloud object *)
Out[6]=

An object outside the list of the ImageNet classes will be misidentified:

In[7]:=
CloudGet["https://www.wolframcloud.com/objects/628da1a9-c2c2-4803-b235-d5c47256ed3b"] (* Evaluate this cell to copy the example input from a cloud object *)
Out[7]=

Obtain the list of names of all available classes:

In[8]:=
EntityValue[
 NetExtract[
   NetModel["ResNet-101 Trained on ImageNet Competition Data"], 
   "Output"][["Labels"]], "Name"]
Out[8]=

Feature extraction

Remove the last three layers of the trained net so that the net produces a vector representation of an image:

In[9]:=
extractor = 
 Take[NetModel[
   "ResNet-101 Trained on ImageNet Competition Data"], {1, -4}]
Out[9]=

Get a set of images:

In[10]:=
CloudGet["https://www.wolframcloud.com/objects/ffd03c1e-d97a-46ba-b98f-215cb773fb73"] (* Evaluate this cell to copy the example input from a cloud object *)

Visualize the features of a set of images:

In[11]:=
FeatureSpacePlot[imgs, FeatureExtractor -> extractor, 
 LabelingFunction -> (ImageResize[#, 100] &), ImageSize -> 800]
Out[11]=

Visualize convolutional weights

Extract the weights of the first convolutional layer in the trained net:

In[12]:=
weights = 
  NetExtract[
   NetModel[
    "ResNet-101 Trained on ImageNet Competition Data"], {"conv1", 
    "Weights"}];

Show the dimensions of the weights:

In[13]:=
Dimensions[weights]
Out[13]=

Visualize the weights as a list of 64 images of size 7x7:

In[14]:=
ImageAdjust[Image[#, Interleaving -> False]] & /@ weights
Out[14]=

Transfer learning

Use the pre-trained model to build a classifier for telling apart images of dogs and cats. Create a test set and a training set:

In[15]:=
CloudGet["https://www.wolframcloud.com/objects/a38111a5-21f7-4f23-8626-a617403ebff5"] (* Evaluate this cell to copy the example input from a cloud object *)
In[16]:=
CloudGet["https://www.wolframcloud.com/objects/c672252d-2eaf-404e-bea9-a9a38b8d46d3"] (* Evaluate this cell to copy the example input from a cloud object *)

Remove the linear layer from the pre-trained net:

In[17]:=
tempNet = 
 Take[NetModel[
   "ResNet-101 Trained on ImageNet Competition Data"], {1, -4}]
Out[17]=

Create a new net composed of the pre-trained net followed by a linear layer and a softmax layer:

In[18]:=
newNet = NetChain[<|"pretrainedNet" -> tempNet, 
   "linearNew" -> LinearLayer[], "softmax" -> SoftmaxLayer[]|>, 
  "Output" -> NetDecoder[{"Class", {"cat", "dog"}}]]
Out[18]=

Train on the dataset, freezing all the weights except for those in the "linearNew" layer (use TargetDevice -> "GPU" for training on a GPU):

In[19]:=
trainedNet = 
 NetTrain[newNet, trainSet, 
  LearningRateMultipliers -> {"linearNew" -> 1, _ -> 0}]
Out[19]=

Perfect accuracy is obtained on the test set:

In[20]:=
ClassifierMeasurements[trainedNet, testSet, "Accuracy"]
Out[20]=

Net information

Inspect the number of parameters of all arrays in the net:

In[21]:=
NetInformation[
 NetModel["ResNet-101 Trained on ImageNet Competition Data"], \
"ArraysElementCounts"]
Out[21]=

Obtain the total number of parameters:

In[22]:=
NetInformation[
 NetModel["ResNet-101 Trained on ImageNet Competition Data"], \
"ArraysTotalElementCount"]
Out[22]=

Obtain the layer type counts:

In[23]:=
NetInformation[
 NetModel["ResNet-101 Trained on ImageNet Competition Data"], \
"LayerTypeCounts"]
Out[23]=

Display the summary graphic:

In[24]:=
NetInformation[
 NetModel["ResNet-101 Trained on ImageNet Competition Data"], \
"SummaryGraphic"]
Out[24]=

Export to MXNet

Export the net into a format that can be opened in MXNet:

In[25]:=
jsonPath = 
 Export[FileNameJoin[{$TemporaryDirectory, "net.json"}], 
  NetModel["ResNet-101 Trained on ImageNet Competition Data"], 
  "MXNet"]
Out[25]=

Export also creates a net.params file containing parameters:

In[26]:=
paramPath = FileNameJoin[{DirectoryName[jsonPath], "net.params"}]
Out[26]=

Get the size of the parameter file:

In[27]:=
FileByteCount[paramPath]
Out[27]=

The size is similar to the byte count of the resource object:

In[28]:=
ResourceObject[
  "ResNet-101 Trained on ImageNet Competition Data"]["ByteCount"]
Out[28]=

Requirements

Wolfram Language 11.2 (September 2017) or above

Reference