Wolfram Computation Meets Knowledge

Dilated ResNet-105 Trained on Cityscapes Data

Segment an image of a driving scenario into semantic component classes

Released in 2017, this architecure combines the technique of dilated convolutions with the paradigm of residual networks, outperforming their nonrelated counterparts in image classification and semantic segmentation.

Number of layers: 360 | Parameter count: 54,497,859 | Trained size: 220 MB

Training Set Information

Performance

Examples

Resource retrieval

Retrieve the resource object:

In[1]:=
ResourceObject["Dilated ResNet-105 Trained on Cityscapes Data"]
Out[1]=

Get the pre-trained net:

In[2]:=
NetModel["Dilated ResNet-105 Trained on Cityscapes Data"]
Out[2]=

Evaluation function

Write an evaluation function to handle net reshaping and resampling of input and output:

In[3]:=
netevaluate[img_, device_: "CPU"] := Block[ {net, encData, dec, mean, var, prob}, net = NetModel["Dilated ResNet-105 Trained on Cityscapes Data"]; encData = Normal@NetExtract[net, "input_0"]; dec = NetExtract[net, "Output"]; {mean, var} = Lookup[encData, {"MeanImage", "VarianceImage"}]; NetReplacePart[net, {"input_0" -> NetEncoder[{"Image", ImageDimensions@img, "MeanImage" -> mean, "VarianceImage" -> var}], "Output" -> dec} ][img, TargetDevice -> device] ]

Label list

Obtain the label list for this model. Integers in the model’s output correspond to elements in the label list:

In[4]:=
labels = {"road", "sidewalk", "building", "wall", "fence", "pole", "traffic light", "traffic sign", "vegetation", "terrain", "sky", "person", "rider", "car", "truck", "bus", "train", "motorcycle", "bicycle"};

Basic usage

Obtain a segmentation mask for a given image:

In[5]:=
CloudGet["https://www.wolframcloud.com/objects/54df76f3-e7e3-47bc-8833-7ee26ba7eda3"] (* Evaluate this cell to copy the example input from a cloud object *)

Inspect which classes are detected:

In[6]:=
detected = DeleteDuplicates@Flatten@mask
Out[6]=
In[7]:=
labels[[detected]]
Out[7]=

Visualize the mask:

In[8]:=
Colorize[mask]
Out[8]=

Advanced visualization

Associate classes to colors:

In[9]:=
colors = Apply[ RGBColor, {{128, 64, 128}, {244, 35, 232}, {70, 70, 70}, {102, 102, 156}, {190, 153, 153}, {153, 153, 153}, {250, 170, 30}, {220, 220, 0}, {107, 142, 35}, {152, 251, 152}, {70, 130, 180}, {220, 20, 60}, {255, 0, 0}, {0, 0, 142}, {0, 0, 70}, {0, 60, 100}, {0, 80, 100}, {0, 0, 230}, {119, 11, 32}}/255., {1}]
Out[9]=
In[10]:=
indexToColor = Thread[Range[19] -> colors];

Write a function to overlap the image and the mask with a legend:

In[11]:=
result[img_, device_: "CPU"] := Block[ {mask, classes, maskPlot, composition}, mask = netevaluate[img, device]; classes = DeleteDuplicates[Flatten@mask]; maskPlot = Colorize[mask, ColorRules -> indexToColor]; composition = ImageCompose[img, {maskPlot, 0.5}]; Legended[ Row[Image[#, ImageSize -> Large] & /@ {maskPlot, composition}], SwatchLegend[indexToColor[[classes, 2]], labels[[classes]]]] ]

Inspect the results:

In[12]:=
CloudGet["https://www.wolframcloud.com/objects/081ec20f-8ee6-4add-815e-ef9be72278da"] (* Evaluate this cell to copy the example input from a cloud object *)
Out[12]=
In[13]:=
CloudGet["https://www.wolframcloud.com/objects/12bdcf7b-4310-44ec-a200-65648262434c"] (* Evaluate this cell to copy the example input from a cloud object *)
Out[13]=
In[14]:=
CloudGet["https://www.wolframcloud.com/objects/cdbe69c4-8765-4148-ad07-a39581ba6093"] (* Evaluate this cell to copy the example input from a cloud object *)
Out[14]=

Net information

Inspect the number of parameters of all arrays in the net:

In[15]:=
NetInformation[ NetModel["Dilated ResNet-105 Trained on Cityscapes Data"], \ "ArraysElementCounts"]
Out[15]=

Obtain the total number of parameters:

In[16]:=
NetInformation[ NetModel["Dilated ResNet-105 Trained on Cityscapes Data"], \ "ArraysTotalElementCount"]
Out[16]=

Obtain the layer type counts:

In[17]:=
NetInformation[ NetModel["Dilated ResNet-105 Trained on Cityscapes Data"], \ "LayerTypeCounts"]
Out[17]=

Export to MXNet

Export the net into a format that can be opened in MXNet:

In[18]:=
jsonPath = Export[FileNameJoin[{$TemporaryDirectory, "net.json"}], NetModel["Dilated ResNet-105 Trained on Cityscapes Data"], "MXNet"]
Out[18]=

Export also creates a net.params file containing parameters:

In[19]:=
paramPath = FileNameJoin[{DirectoryName[jsonPath], "net.params"}]
Out[19]=

Get the size of the parameter file:

In[20]:=
FileByteCount[paramPath]
Out[20]=

The size is similar to the byte count of the resource object:

In[21]:=
ResourceObject[ "Dilated ResNet-105 Trained on Cityscapes Data"]["ByteCount"]
Out[21]=

Requirements

Wolfram Language 11.3 (March 2018) or above

Reference