Wolfram Computation Meets Knowledge

Pix2pix Photo-to-Street-Map Translation

Generate a street map from a satellite photo

Released in 2016, this model is an application of a powerful method for general-purpose image-to-image translation using conditional adversarial networks. The automatic learning of the loss function with the adversarial networks technique allows the same paradigm to generalize across a wide range of image translation tasks. The architecture enables an efficient aggregation of features of multiple scales through skip connections with concatenations. This particular model was trained to generate a street map from a satellite photo.

Number of layers: 56 | Parameter count: 54,419,459 | Trained size: 218 MB

Training Set Information

Examples

Resource retrieval

Get the pre-trained net:

In[1]:=
NetModel["Pix2pix Photo-to-Street-Map Translation"]
Out[2]=

Basic usage

Obtain a satellite photo:

In[3]:=
img = ImageResize[ GeoImage[Entity["City", {"LasVegas", "Nevada", "UnitedStates"}], GeoRange -> 200], {256, 256}]
Out[3]=

Use the net to draw the street map:

In[4]:=
map = NetModel["Pix2pix Photo-to-Street-Map Translation"][img]
Out[5]=

Evaluate accuracy

Overlap photo and prediction:

In[6]:=
ImageCompose[img, SetAlphaChannel[map, 0.8]]
Out[6]=

Obtain the actual street map:

In[7]:=
actualMap = ImageResize[ GeoImage[Entity["City", {"LasVegas", "Nevada", "UnitedStates"}], "StreetMap", GeoRange -> 200], {256, 256}]
Out[7]=

Compare the generated street map with the actual street map:

In[8]:=
ImageAssemble[{map, actualMap}]
Out[8]=

Issues

More complex patterns are harder to render. Obtain a new photo and street map pair:

In[9]:=
{img, actualMap} = ImageResize[ GeoImage[GeoPosition[{41.940360, 12.533762}], #, GeoRange -> 200], {256, 256}] & /@ {"Satellite", "StreetMap"}
Out[9]=

Compare the prediction with the actual street map:

In[10]:=
ImageAssemble[{NetModel["Pix2pix Photo-to-Street-Map Translation"][ img], actualMap}]
Out[11]=

Net information

Inspect the number of parameters of all arrays in the net:

In[12]:=
NetInformation[ NetModel["Pix2pix Photo-to-Street-Map Translation"], \ "ArraysElementCounts"]
Out[107]=

Obtain the total number of parameters:

In[108]:=
NetInformation[ NetModel["Pix2pix Photo-to-Street-Map Translation"], \ "ArraysTotalElementCount"]
Out[109]=

Obtain the layer type counts:

In[110]:=
NetInformation[ NetModel["Pix2pix Photo-to-Street-Map Translation"], \ "LayerTypeCounts"]
Out[111]=

Display the summary graphic:

In[112]:=
NetInformation[ NetModel["Pix2pix Photo-to-Street-Map Translation"], \ "SummaryGraphic"]
Out[113]=

Export to MXNet

Export the net into a format that can be opened in MXNet:

In[114]:=
jsonPath = Export[FileNameJoin[{$TemporaryDirectory, "net.json"}], NetModel["Pix2pix Photo-to-Street-Map Translation"], "MXNet"]
Out[115]=

Export also creates a net.params file containing parameters:

In[116]:=
paramPath = FileNameJoin[{DirectoryName[jsonPath], "net.params"}]
Out[116]=

Get the size of the parameter file:

In[117]:=
FileByteCount[paramPath]
Out[117]=

The size is similar to the byte count of the resource object:

In[118]:=
ResourceObject["Pix2pix Photo-to-Street-Map Translation"]["ByteCount"]
Out[119]=

Represent the MXNet net as a graph:

In[120]:=
Import[jsonPath, {"MXNet", "NodeGraphPlot"}]
Out[120]=

Requirements

Wolfram Language 11.2 (September 2017) or above

Resource History

  • Date Created: 12 October 2017
  • Latest Update: 21 June 2018

Reference