Wolfram Computation Meets Knowledge

Unguided Volumetric Regression Net for 3D Face Reconstruction

Reconstruct a 3D facial image from a 2D facial image

Released in 2017, this net tackles the problem of regressing a 3D facial model in an end-to-end fashion, bypassing many of the difficulties found in complex pipelines involving multiple stages. Starting from a single input image, the facial model is directly reconstructed as a three-dimensional array of pixel intensities. Its architecture is based on the "hourglass" design.

Number of layers: 1,029 | Parameter count: 19,277,384 | Trained size: 81 MB

Training Set Information

Examples

Resource retrieval

Retrieve the resource object:

In[1]:=
ResourceObject["Unguided Volumetric Regression Net for 3D Face \
Reconstruction"]
Out[1]=

Get the pre-trained net:

In[2]:=
NetModel["Unguided Volumetric Regression Net for 3D Face \
Reconstruction"]
Out[2]=

Basic usage

Get the volumetric model of a facial image:

In[3]:=

Obtain the dimensions of the volumetric model:

In[4]:=
Dimensions[face3D]
Out[4]=

Visualize the volumetric representation:

In[5]:=
img3D = Image3D[255*face3D, "Byte", BoxRatios -> {1, 1, 0.5}, 
  ViewPoint -> Below]
Out[5]=

Extract a 3D mesh:

In[6]:=
ImageMesh[img3D, Method -> "DualMarchingCubes", 
 BoxRatios -> {1, 1, 0.5}, ViewPoint -> Below]
Out[6]=

Robustness to facial crop size

Get an image:

In[7]:=

Crop the image at various sizes:

In[8]:=
crops = Table[ImageCrop[img, s, Bottom], {s, 290, 500, 70}]
Out[8]=

Inspect the network performance across the crops:

In[9]:=
Image3D[255*
    NetModel[
      "Unguided Volumetric Regression Net for 3D Face \
Reconstruction"][#], "Byte", BoxRatios -> {1, 1, 0.5}, 
   ViewPoint -> Below, ImageSize -> Small] & /@ crops
Out[9]=

Export to MXNet

Export the net into a format that can be opened in MXNet:

In[10]:=
jsonPath = 
 Export[FileNameJoin[{$TemporaryDirectory, "net.json"}], 
  NetModel["Unguided Volumetric Regression Net for 3D Face \
Reconstruction"], "MXNet"]
Out[10]=

Export also creates a net.params file containing parameters:

In[11]:=
paramPath = FileNameJoin[{DirectoryName[jsonPath], "net.params"}]
Out[11]=

Get the size of the parameter file:

In[12]:=
FileByteCount[paramPath]
Out[12]=

The size is similar to the byte count of the resource object:

In[13]:=
ResourceObject[
  "Unguided Volumetric Regression Net for 3D Face \
Reconstruction"]["ByteCount"]
Out[13]=

Requirements

Wolfram Language 11.2 (September 2017) or above

Reference

  • A. S. Jackson, A. Bulat, V. Argyriou, G. Tzimiropoulos, "Large Pose 3D Face Reconstruction from a Single Image via Direct Volumetric CNN Regression," arXiv:1703.07834 (2017)
  • (available from https://github.com/AaronJackson/vrn)
  • Rights: MIT License