Resource retrieval

Get the pre-trained net:

In[1]:=

Out[1]=

NetModel parameters

This model consists of a family of individual nets, each identified by a specific parameter combination. Inspect the available parameters:

In[2]:=

Out[2]=

Pick a non-default net by specifying the parameters:

In[3]:=

Out[3]=

Pick a non-default uninitialized net:

In[4]:=

Out[4]=

Pick the tokenizer:

In[5]:=

Out[5]=

Evaluation function

Write an evaluation function to automate the image features computation and the text generation loop, directly producing the caption:

In[6]:=

Basic usage

Define a test image:

In[7]:=

Generate an image caption:

In[8]:=

Out[8]=

Try different initial prompts:

In[9]:=

Out[9]=

Obtain a test video:

In[10]:=

Out[2]=

Generate a caption for the video. The caption is generated from a number of uniformly spaced frames whose features are averaged. The number of frames can be controlled via an option:

In[11]:=

Out[11]=

Feature space visualization

Get a set of images of cars and airplanes:

In[12]:=

Visualize the feature space embedding performed by the image encoder. Notice that images from the same class are clustered together:

In[13]:=

Out[13]=

Cross-attention visualization

When generating a new token of the image caption, the text decoder attends on the image features produced by the image encoder. Such features are a set of 577 vectors of length 1024, where every vector except the first one corresponds to one of 24x24 patches taken from the input image (the extra vector exists because the image encoder inherits the architecture of the image classification model Vision Transformer Trained on ImageNet Competition Data, but in this case, it doesn’t have any special importance). This means that the text decoder attention weights to these image features can be interpreted as the image patches the decoder is "looking at" when generating every new token, and it is possible to visualize this information. Get a test image and compute the features:

In[14]:=

In[15]:=

In[16]:=

Out[16]=

Generate the caption, accumulating the attention weights for each token generation. There are 12 attention blocks in the text decoder and each generates its own set of attention weights, and as usual in deep learning, the deeper blocks are the ones generating the most semantic information. Hence the following code extracts the attention weights from the last attention block only:

In[17]:=

Out[31]=

Each generated token corresponds to a 12x577 array of attention weights, where 12 is the number of attention heads and 577 is the 24x24 patches plus the extra one:

In[32]:=

Out[32]=

Extract the attention weights related to the image patches and rescale them by their minimum value, which makes them more suitable for visualization:

In[33]:=

In[34]:=

Out[34]=

Reshape the flat image patch dimension to 24x24 and take the average over the attention heads, thus obtaining a 24x24 attention matrix for each of the 10 generated tokens:

In[35]:=

In[36]:=

Out[36]=

Visualize the attention weight matrices. Patches with higher values (red) are what is mostly being "looked at" when generating the corresponding token:

In[37]:=

Out[37]=

Define a function to visualize the attention matrix on the image:

In[38]:=

Visualize the attention mechanism for each token. Notice the emphasis on the head for the tokens "little" and "girl," on the hands for "taking" and on the camera for "camera":

In[39]:=

In[40]:=

Out[40]=

Net information

Inspect the number of parameters of all arrays in the net:

In[41]:=

Out[41]=

Obtain the total number of parameters:

In[42]:=

Out[42]=

Obtain the layer type counts:

In[43]:=

Out[43]=