Wolfram Computation Meets Knowledge

Single-Image Depth Perception Net Trained on NYU Depth V2 Data

Estimate the depth map of an image

Released in 2016, this neural net was trained to predict the relative depth map from a single image using a novel technique based on sparse ordinal annotations. Each training example only needs to be annotated with a pair of points and its relative distance to the camera. After training, the net is able to reconstruct the full depth map. Its architecture is based on the "hourglass" design.

Number of layers: 501 | Parameter count: 5,385,185 | Trained size: 23 MB

Training Set Information

Examples

Resource retrieval

Retrieve the resource object:

In[1]:=
ResourceObject["Single-Image Depth Perception Net Trained on NYU \
Depth V2 Data"]
Out[1]=

Get the pre-trained net:

In[2]:=
NetModel["Single-Image Depth Perception Net Trained on NYU Depth V2 \
Data"]
Out[2]=

Basic usage

Obtain the depth map of an image:

In[3]:=
Out[3]=

Obtain the dimensions of the depth map:

In[4]:=
Dimensions[depthMap]
Out[4]=

Show the depth map:

In[5]:=
ImageAdjust[Image[depthMap]]
Out[5]=

Visualize a 3D model

Get an image:

In[6]:=

Obtain the depth map:

In[7]:=
depthMap = 
 NetModel["Single-Image Depth Perception Net Trained on NYU Depth V2 \
Data"][img]
Out[7]=

Visualize a 3D model using the depth map:

In[8]:=
ListPlot3D[-depthMap, PlotStyle -> Texture[ImageReflect[img]], 
 PlotTheme -> {"Minimal", "NoAxes"}, 
 ViewPoint -> {0, Sqrt[2], Sqrt[2]}]
Out[8]=

Adapt to any size

The recommended way to deal with image sizes and aspect ratios is to resample the depth map after the net evaluation. Get an image:

In[9]:=

Obtain the dimensions of the image:

In[10]:=
ImageDimensions[img]
Out[10]=

Obtain the depth map

In[11]:=

Obtain the dimensions of the depth map:

In[12]:=
Dimensions[depthMap]
Out[12]=

Resample the depth map and visualize it:

In[13]:=
ImageAdjust[
 Image[ArrayResample[depthMap, Reverse[ImageDimensions[img]]]]]
Out[13]=

Export to MXNet

Export the net into a format that can be opened in MXNet:

In[14]:=
jsonPath = 
 Export[FileNameJoin[{$TemporaryDirectory, "net.json"}], 
  NetModel["Single-Image Depth Perception Net Trained on NYU Depth V2 \
Data"], "MXNet"]
Out[14]=

Export also creates a net.params file containing parameters:

In[15]:=
paramPath = FileNameJoin[{DirectoryName[jsonPath], "net.params"}]
Out[15]=

Get the size of the parameter file:

In[16]:=
FileByteCount[paramPath]
Out[16]=

The size is similar to the byte count of the resource object:

In[17]:=
ResourceObject[
  "Single-Image Depth Perception Net Trained on NYU Depth V2 \
Data"]["ByteCount"]
Out[17]=

Requirements

Wolfram Language 11.2 (September 2017) or above

Reference