Single-Image Depth Perception Net Trained on Depth in the Wild Data

Estimate the depth map of an image

Released in 2016, this neural net was trained to predict the relative depth map from a single image using a novel technique based on sparse ordinal annotations. Each training example only needs to be annotated with a pair of points and its relative distance to the camera. After training, the net is able to reconstruct the full depth map. Its architecture is based on the "hourglass" design.

Number of layers: 501 | Parameter count: 5,385,185 | Trained size: 23 MB |

Training Set Information

Depth in the Wild, consisting of 495,000 diverse images, each annotated with a randomly sampled pair of points and their relative depth.

Performance

This model achieves 22.14% WHDR (Weighted Human Disagreement Rate) error on the Depth in the Wild dataset with all weights set to 1.

Examples

Download Example Notebook

Open in Wolfram Cloud

Resource retrieval

Get the pre-trained net:

In[1]:=

$NetModel["Single-Image Depth Perception Net Trained on Depth in the \ Wild Data"]$

Out[1]=

Basic usage

Obtain the depth map of an image:

In[2]:=

(* Evaluate this cell to get the example input *) CloudGet["https://www.wolframcloud.com/obj/02b6c29e-3231-48e7-be51-38169cd5f982"]

Out[2]=

Show the depth map:

In[3]:=

Out[3]=

Visualize a 3D model

Get an image:

In[4]:=

(* Evaluate this cell to get the example input *) CloudGet["https://www.wolframcloud.com/obj/bba6747a-a183-4440-93d0-0e71587df7ed"]

In[5]:=

Obtain the depth map:

In[6]:=

$depthMap = NetModel["Single-Image Depth Perception Net Trained on Depth in the \ Wild Data"][img]$

Out[6]=

Visualize a 3D model using the depth map:

In[7]:=

Out[7]=

Adapt to any size

The recommended way to deal with image sizes and aspect ratios is to resample the depth map after the net evaluation. Get an image:

In[8]:=

(* Evaluate this cell to get the example input *) CloudGet["https://www.wolframcloud.com/obj/f69a8a3b-a75a-4ab4-8b2a-cdc1d5e12f62"]

Obtain the dimensions of the image:

In[9]:=

Out[9]=

Obtain the depth map

In[10]:=

$depthMap = NetModel["Single-Image Depth Perception Net Trained on Depth in the \ Wild Data"][img]$

Out[10]=

Resample the depth map and visualize it:

In[11]:=

Out[11]=

Net information

Inspect the number of parameters of all arrays in the net:

In[12]:=

$NetInformation[ NetModel["Single-Image Depth Perception Net Trained on Depth in the \ Wild Data"], "ArraysElementCounts"]$

Out[12]=

Obtain the total number of parameters:

In[13]:=

$NetInformation[ NetModel["Single-Image Depth Perception Net Trained on Depth in the \ Wild Data"], "ArraysTotalElementCount"]$

Out[13]=

Obtain the layer type counts:

In[14]:=

$NetInformation[ NetModel["Single-Image Depth Perception Net Trained on Depth in the \ Wild Data"], "LayerTypeCounts"]$

Out[14]=

Display the summary graphic:

In[15]:=

$NetInformation[ NetModel["Single-Image Depth Perception Net Trained on Depth in the \ Wild Data"], "SummaryGraphic"]$

Out[15]=

Export to MXNet

Export the net into a format that can be opened in MXNet:

In[16]:=

$jsonPath = Export[FileNameJoin[{$TemporaryDirectory, "net.json"}], NetModel["Single-Image Depth Perception Net Trained on Depth in the \ Wild Data"], "MXNet"]$

Out[16]=

Export also creates a net.params file containing parameters:

In[17]:=

Out[17]=

Get the size of the parameter file:

In[18]:=

Out[18]=

The size is similar to the byte count of the resource object:

In[19]:=

$ResourceObject[ "Single-Image Depth Perception Net Trained on Depth in the Wild \ Data"]["ByteCount"]$

Out[19]=

Construction Notebook

Download Construction Notebook

Open in Wolfram Cloud

Requirements

Wolfram Language 11.2 (September 2017) or above

Resource History

Date Created: 13 October 2017
Latest Update: 21 June 2018

Reference

W. Chen, Z. Fu, D. Yang and J. Deng, "Single-Image Depth Perception in the Wild," arXiv:1604.03901 (2016)
Available from: https://github.com/wfchen-umich/relative_depth
Rights: BSD 3-Clause License