Wolfram Research

Unguided Volumetric Regression Net for 3D Face Reconstruction

Reconstruct a 3D facial image from a 2D facial image

Released in 2017, this net tackles the problem of regressing a 3D facial model in an end-to-end fashion, bypassing many of the difficulties found in complex pipelines involving multiple stages. Starting from a single input image, the facial model is directly reconstructed as a three-dimensional array of pixel intensities. Its architecture is based on the "hourglass" design.

Number of layers: 1,029 | Parameter count: 19,277,384 | Trained size: 81 MB

Training Set Information

Performance

Examples

Resource retrieval

Get the pre-trained net:

In[1]:=
NetModel["Unguided Volumetric Regression Net for 3D Face \
Reconstruction"]
Out[1]=

Basic usage

Get the volumetric model of a facial image:

In[2]:=
CloudGet["https://www.wolframcloud.com/objects/b14bc770-5d09-40b1-a4fb-19c9de229548"] (* Evaluate this cell to copy the example input from a cloud object *)

Visualize the volumetric representation:

In[3]:=
img3D = Image3D[255*face3D, "Byte", BoxRatios -> {1, 1, 0.5}, 
  ViewPoint -> Below]
Out[3]=

Extract a 3D mesh:

In[4]:=
ImageMesh[img3D, Method -> "DualMarchingCubes", 
 BoxRatios -> {1, 1, 0.5}, ViewPoint -> Below]
Out[4]=

Robustness to facial crop size

Get an image:

In[5]:=
CloudGet["https://www.wolframcloud.com/objects/23dc65dd-d946-4b69-b972-a78cf8c3a935"] (* Evaluate this cell to copy the example input from a cloud object *)

Crop the image at various sizes:

In[6]:=
crops = Table[ImageCrop[img, s, Bottom], {s, 290, 500, 70}]
Out[6]=

Inspect the Net performance across the crops:

In[7]:=
Image3D[255*
    NetModel[
      "Unguided Volumetric Regression Net for 3D Face \
Reconstruction"][#], "Byte", BoxRatios -> {1, 1, 0.5}, 
   ViewPoint -> Below, ImageSize -> Small] & /@ crops
Out[7]=

Net information

Inspect the number of parameters of all arrays in the net:

In[8]:=
NetInformation[
 NetModel["Unguided Volumetric Regression Net for 3D Face \
Reconstruction"], "ArraysElementCounts"]
Out[8]=

Obtain the total number of parameters:

In[9]:=
NetInformation[
 NetModel["Unguided Volumetric Regression Net for 3D Face \
Reconstruction"], "ArraysTotalElementCount"]
Out[9]=

Obtain the layer type counts:

In[10]:=
NetInformation[
 NetModel["Unguided Volumetric Regression Net for 3D Face \
Reconstruction"], "LayerTypeCounts"]
Out[10]=

Display the summary graphic:

In[11]:=
NetInformation[
 NetModel["Unguided Volumetric Regression Net for 3D Face \
Reconstruction"], "SummaryGraphic"]
Out[11]=

Export to MXNet

Export the net into a format that can be opened in MXNet:

In[12]:=
jsonPath = 
 Export[FileNameJoin[{$TemporaryDirectory, "net.json"}], 
  NetModel["Unguided Volumetric Regression Net for 3D Face \
Reconstruction"], "MXNet"]
Out[12]=

Export also creates a net.params file containing parameters:

In[13]:=
paramPath = FileNameJoin[{DirectoryName[jsonPath], "net.params"}]
Out[13]=

Get the size of the parameter file:

In[14]:=
FileByteCount[paramPath]
Out[14]=

The size is similar to the byte count of the resource object:

In[15]:=
ResourceObject[
  "Unguided Volumetric Regression Net for 3D Face \
Reconstruction"]["ByteCount"]
Out[15]=

Requirements

Wolfram Language 11.2 (September 2017) or above

Resource History

Reference

  • A. S. Jackson, A. Bulat, V. Argyriou, G. Tzimiropoulos, "Large Pose 3D Face Reconstruction from a Single Image via Direct Volumetric CNN Regression," arXiv:1703.07834 (2017)
  • (available from https://github.com/AaronJackson/vrn)
  • Rights: MIT License