Wolfram Neural Net Repository

Immediate Computable Access to Neural Net Models

AdaIN-Style Trained on MS-COCO and Painter by Numbers Data

Transfer the style of one image to another image

Released in 2017, this is the first real-time feedforward image stylization model to accept arbitrary styles. Building on the interpretation of neural style transfer as a statistical domain adaptation task, the model leverages a novel technique called Adaptive Instance Normalization (AdaIN). The AdaIN layer inside the net performs the style transfer by aligning the mean and variance of the content and style feature maps.

Number of layers: 109 | Parameter count: 10,516,675 | Trained size: 42 MB |

Training Set Information

Examples

Resource retrieval

Get the pre-trained net:

In[1]:=
NetModel["AdaIN-Style Trained on MS-COCO and Painter by Numbers Data"]
Out[1]=

Basic usage

Restyle an image:

In[2]:=
(* Evaluate this cell to get the example input *) CloudGet["https://www.wolframcloud.com/obj/2e847ada-28ed-4289-ac98-9948f88a3f11"]
Out[2]=

Control the stylization weighting

More flexibility can be obtained by manipulating the different subnetworks separately. Separate the NetGraph into its components:

In[3]:=
net = NetModel[ "AdaIN-Style Trained on MS-COCO and Painter by Numbers Data"]
Out[3]=
In[4]:=
netEncoder = NetExtract[net, "Content"]
Out[4]=
In[5]:=
encoderChain = NetExtract[net, "encoder_content"]
Out[5]=
In[6]:=
adaIN = NetExtract[net, "adaIN"]
Out[6]=
In[7]:=
decoderChain = NetExtract[net, "decoder"]
Out[7]=
In[8]:=
netDecoder = NetExtract[net, "Output"]
Out[8]=

Obtain the encoded content and style features:

In[9]:=
(* Evaluate this cell to get the example input *) CloudGet["https://www.wolframcloud.com/obj/ff33c260-d665-4a6c-830e-460d4f73c0d4"]
In[10]:=
(* Evaluate this cell to get the example input *) CloudGet["https://www.wolframcloud.com/obj/2e7da7ac-d193-46bf-8b11-83171744bbe0"]

Run the net on the features:

In[11]:=
adaptedFeatures = adaIN[<|"c_in" -> contentFeatures, "s_in" -> styleFeatures|>];

To control the stylization weight, blend the adapted features with the content features:

In[12]:=
blendedFeatures[w_] := w*adaptedFeatures + (1 - w)*contentFeatures

Stylize the content with different style weights:

In[13]:=
netDecoder@decoderChain[blendedFeatures[0.3]]
Out[13]=
In[14]:=
netDecoder@decoderChain[blendedFeatures[0.9]]
Out[14]=

Adapt to any size

Automatic image resizing can be avoided by replacing the net encoders. First get the net:

In[15]:=
net = NetModel[ "AdaIN-Style Trained on MS-COCO and Painter by Numbers Data"]
Out[15]=

Extract the original mean channel values:

In[16]:=
meanRGB = NetExtract[net, "Content"][["MeanImage"]]
Out[16]=

Obtain the content and style features:

In[17]:=
(* Evaluate this cell to get the example input *) CloudGet["https://www.wolframcloud.com/obj/4225d1eb-bed4-46cc-960c-57a8fe408cf9"]
In[18]:=
(* Evaluate this cell to get the example input *) CloudGet["https://www.wolframcloud.com/obj/5f1f3ae4-1e2b-4652-b0fd-c50e116314ac"]

Create the new net encoders with the desired dimensions:

In[19]:=
netEncC = NetEncoder[{"Image", ImageDimensions[content], "MeanImage" -> meanRGB}];
In[20]:=
netEncS = NetEncoder[{"Image", ImageDimensions[style], "MeanImage" -> meanRGB}];

Attach the new net encoders:

In[21]:=
resizedNet = NetReplacePart[ net, {"Content" -> netEncC, "Style" -> netEncS, "Output" -> NetDecoder["Image"]}]
Out[21]=

Restyle an image using the new resized network:

In[22]:=
resizedNet@<|"Content" -> content, "Style" -> style|>
Out[22]=

Net information

Inspect the number of parameters of all arrays in the net:

In[23]:=
NetInformation[ NetModel["AdaIN-Style Trained on MS-COCO and Painter by Numbers \ Data"], "ArraysElementCounts"]
Out[23]=

Obtain the total number of parameters:

In[24]:=
NetInformation[ NetModel["AdaIN-Style Trained on MS-COCO and Painter by Numbers \ Data"], "ArraysTotalElementCount"]
Out[24]=

Obtain the layer type counts:

In[25]:=
NetInformation[ NetModel["AdaIN-Style Trained on MS-COCO and Painter by Numbers \ Data"], "LayerTypeCounts"]
Out[25]=

Display the summary graphic:

In[26]:=
NetInformation[ NetModel["AdaIN-Style Trained on MS-COCO and Painter by Numbers \ Data"], "SummaryGraphic"]
Out[26]=

Export to MXNet

Export the net into a format that can be opened in MXNet:

In[27]:=
jsonPath = Export[FileNameJoin[{$TemporaryDirectory, "net.json"}], NetModel["AdaIN-Style Trained on MS-COCO and Painter by Numbers \ Data"], "MXNet"]
Out[27]=

Export also creates a net.params file containing parameters:

In[28]:=
paramPath = FileNameJoin[{DirectoryName[jsonPath], "net.params"}]
Out[28]=

Get the size of the parameter file:

In[29]:=
FileByteCount[paramPath]
Out[29]=

The size is similar to the byte count of the resource object:

In[30]:=
ResourceObject[ "AdaIN-Style Trained on MS-COCO and Painter by Numbers \ Data"]["ByteCount"]
Out[30]=

Represent the MXNet net as a graph:

In[31]:=
Import[jsonPath, {"MXNet", "NodeGraphPlot"}]
Out[31]=

Requirements

Wolfram Language 11.2 (September 2017) or above

Resource History

  • Date Created: 19 July 2017
  • Latest Update: 30 July 2021

Reference