Wolfram Computation Meets Knowledge

Unguided Volumetric Regression Net for 3D Face Reconstruction

Reconstruct a 3D facial image from a 2D facial image

Released in 2017, this net tackles the problem of regressing a 3D facial model in an end-to-end fashion, bypassing many of the difficulties found in complex pipelines involving multiple stages. Starting from a single input image, the facial model is directly reconstructed as a three-dimensional array of pixel intensities. Its architecture is based on the "hourglass" design.

Number of layers: 1,029 | Parameter count: 19,277,384 | Trained size: 81 MB

Training Set Information

Performance

Examples

Resource retrieval

Retrieve the resource object:

In[1]:=
ResourceObject["Unguided Volumetric Regression Net for 3D Face \
Reconstruction"]
Out[2]=

Get the pre-trained net:

In[3]:=
NetModel["Unguided Volumetric Regression Net for 3D Face \
Reconstruction"]
Out[4]=

Basic usage

Get the volumetric model of a facial image:

In[5]:=
CloudGet["https://www.wolframcloud.com/objects/0371aad8-b36d-4193-896c-d4e8ee36fc34"] (* Evaluate this cell to copy the example input from a cloud object *)

Obtain the dimensions of the volumetric model:

In[6]:=
Dimensions[face3D]
Out[6]=

Visualize the volumetric representation:

In[7]:=
img3D = Image3D[255*face3D, "Byte", BoxRatios -> {1, 1, 0.5}, 
  ViewPoint -> Below]
Out[7]=

Extract a 3D mesh:

In[8]:=
ImageMesh[img3D, Method -> "DualMarchingCubes", 
 BoxRatios -> {1, 1, 0.5}, ViewPoint -> Below]
Out[8]=

Robustness to facial crop size

Get an image:

In[9]:=
CloudGet["https://www.wolframcloud.com/objects/b541f5f1-6b9e-41ba-9f22-6b36a10ab212"] (* Evaluate this cell to copy the example input from a cloud object *)

Crop the image at various sizes:

In[10]:=
crops = Table[ImageCrop[img, s, Bottom], {s, 290, 500, 70}]
Out[10]=

Inspect the Net performance across the crops:

In[11]:=
Image3D[255*
    NetModel[
      "Unguided Volumetric Regression Net for 3D Face \
Reconstruction"][#], "Byte", BoxRatios -> {1, 1, 0.5}, 
   ViewPoint -> Below, ImageSize -> Small] & /@ crops
Out[12]=

Net information

Inspect the number of parameters of all arrays in the net:

In[13]:=
NetInformation[
 NetModel["Unguided Volumetric Regression Net for 3D Face \
Reconstruction"], "ArraysElementCounts"]
Out[13]=

Obtain the total number of parameters:

In[14]:=
NetInformation[
 NetModel["Unguided Volumetric Regression Net for 3D Face \
Reconstruction"], "ArraysTotalElementCount"]
Out[14]=

Obtain the layer type counts:

In[15]:=
NetInformation[
 NetModel["Unguided Volumetric Regression Net for 3D Face \
Reconstruction"], "LayerTypeCounts"]
Out[15]=

Display the summary graphic:

In[16]:=
NetInformation[
 NetModel["Unguided Volumetric Regression Net for 3D Face \
Reconstruction"], "SummaryGraphic"]
Out[16]=

Export to MXNet

Export the net into a format that can be opened in MXNet:

In[17]:=
jsonPath = 
 Export[FileNameJoin[{$TemporaryDirectory, "net.json"}], 
  NetModel["Unguided Volumetric Regression Net for 3D Face \
Reconstruction"], "MXNet"]
Out[17]=

Export also creates a net.params file containing parameters:

In[18]:=
paramPath = FileNameJoin[{DirectoryName[jsonPath], "net.params"}]
Out[18]=

Get the size of the parameter file:

In[19]:=
FileByteCount[paramPath]
Out[19]=

The size is similar to the byte count of the resource object:

In[20]:=
ResourceObject[
  "Unguided Volumetric Regression Net for 3D Face \
Reconstruction"]["ByteCount"]
Out[20]=

Requirements

Wolfram Language 11.2 (September 2017) or above

Reference

  • A. S. Jackson, A. Bulat, V. Argyriou, G. Tzimiropoulos, "Large Pose 3D Face Reconstruction from a Single Image via Direct Volumetric CNN Regression," arXiv:1703.07834 (2017)
  • (available from https://github.com/AaronJackson/vrn)
  • Rights: MIT License