Wolfram Computation Meets Knowledge

3D Face Alignment Net Trained on 300W Large Pose Data

Determine the 2D projection of 3D keypoints from a facial image

Developed in 2017 at the Computer Vision Laboratory at the University of Nottingham, this net predicts the locations of the 2D projections of 68 3D keypoints (17 for face contour, 10 for eyebrows, 9 for nose, 12 for eyes, 20 for mouth) from a facial image. For each keypoint, a heat map for its location is produced. Its complex architecture features a combination of hourglass modules and multiscale parallel blocks.

Number of layers: 967 | Parameter count: 23,874,320 | Trained size: 97 MB

Training Set Information

Performance

Examples

Resource retrieval

Retrieve the resource object:

In[1]:=
ResourceObject["3D Face Alignment Net Trained on 300W Large Pose \
Data"]
Out[1]=

Get the pre-trained net:

In[2]:=
NetModel["3D Face Alignment Net Trained on 300W Large Pose Data"]
Out[2]=

Basic usage

This net outputs a 64x64 heat map for each of the 68 landmarks:

In[3]:=
CloudGet["https://www.wolframcloud.com/objects/87c13c81-923a-4585-a43d-5c2b8afca3ce"] (* Evaluate this cell to copy the example input from a cloud object *)
Out[3]=

Obtain the dimensions of the heat maps:

In[4]:=
Dimensions[heatmaps]
Out[4]=

Visualize heat maps 1, 12 and 29:

In[5]:=
MatrixPlot /@ heatmaps[[{1, 12, 29}]]
Out[5]=

Evaluation function

Write an evaluation function that picks the maximum position of each heat map and returns a list of landmark positions:

In[6]:=
netevaluation[img_] := Block[
  {heatmaps, posFlattened, posMat},
  heatmaps = 
   NetModel["3D Face Alignment Net Trained on 300W Large Pose Data"][
    img];
  posFlattened = 
   Map[First@Ordering[#, -1] &, Flatten[heatmaps, {{1}, {2, 3}}]];
  posMat = QuotientRemainder[posFlattened - 1, 64] + 1;
  1/64.*Map[{#[[2]], 64 - #[[1]] + 1} - 0.5 &, posMat]
  ]

Landmark positions

Get the landmarks using the evaluation function. Coordinates are rescaled to the input image size so that the bottom-left corner is identified by {0, 0} and the top-right corner by {1, 1}:

In[7]:=
CloudGet["https://www.wolframcloud.com/objects/9cc0fa23-8d51-4175-8f23-f57099870f1d"] (* Evaluate this cell to copy the example input from a cloud object *)
Out[7]=

Group landmarks associated with different facial features by colors:

In[8]:=
groupings = 
  Span @@@ {{1, 17}, {18, 22}, {23, 27}, {28, 36}, {37, 42}, {43, 
     48}, {49, 60}, {61, 68}};

Visualize the landmarks:

In[9]:=
CloudGet["https://www.wolframcloud.com/objects/512e4b7d-c0a7-4f6a-8d63-e8338349a942"] (* Evaluate this cell to copy the example input from a cloud object *)
Out[9]=

Robustness to facial crop size

Get an image:

In[10]:=
CloudGet["https://www.wolframcloud.com/objects/4a61bc0d-cd9a-4a33-a51d-30a3a6d90637"] (* Evaluate this cell to copy the example input from a cloud object *)

Crop the image at various sizes:

In[11]:=
crops = Table[ImageCrop[img, s, Bottom], {s, 290, 500, 70}]
Out[11]=

Inspect the network performance across the crops:

In[12]:=
HighlightImage[#, 
   Graphics@
    Riffle[Thread@Hue[Range[8]/8.], 
     Map[Point, Function[p, Part[netevaluation[#], p]] /@ groupings]],
    DataRange -> {{0, 1}, {0, 1}}, ImageSize -> 250] & /@ crops
Out[12]=

Net information

Inspect the number of parameters of all arrays in the net:

In[13]:=
NetInformation[
 NetModel["3D Face Alignment Net Trained on 300W Large Pose Data"], \
"ArraysElementCounts"]
Out[13]=

Obtain the total number of parameters:

In[14]:=
NetInformation[
 NetModel["3D Face Alignment Net Trained on 300W Large Pose Data"], \
"ArraysTotalElementCount"]
Out[14]=

Obtain the layer type counts:

In[15]:=
NetInformation[
 NetModel["3D Face Alignment Net Trained on 300W Large Pose Data"], \
"LayerTypeCounts"]
Out[15]=

Display the summary graphic:

In[16]:=
NetInformation[
 NetModel["3D Face Alignment Net Trained on 300W Large Pose Data"], \
"SummaryGraphic"]
Out[16]=

Export to MXNet

Export the net into a format that can be opened in MXNet:

In[17]:=
jsonPath = 
 Export[FileNameJoin[{$TemporaryDirectory, "net.json"}], 
  NetModel["3D Face Alignment Net Trained on 300W Large Pose Data"], 
  "MXNet"]
Out[17]=

Export also creates a net.params file containing parameters:

In[18]:=
paramPath = FileNameJoin[{DirectoryName[jsonPath], "net.params"}]
Out[18]=

Get the size of the parameter file:

In[19]:=
FileByteCount[paramPath]
Out[19]=

The size is similar to the byte count of the resource object:

In[20]:=
NetModel["3D Face Alignment Net Trained on 300W Large Pose Data", \
"ByteCount"]
Out[20]=

Requirements

Wolfram Language 11.2 (September 2017) or above

Resource History

Reference