Resource retrieval

Get the pre-trained net:

In[1]:=

Out[1]=

NetModel parameters

This model consists of a family of individual nets, each identified by a specific parameter combination. Inspect the available parameters:

In[2]:=

Out[2]=

Pick a non-default net by specifying the parameters:

In[3]:=

Out[3]=

Pick a non-default uninitialized net:

In[4]:=

Out[4]=

Get the labels:

In[5]:=

Out[5]=

Evaluation function

Write an evaluation function to combine the encoder and decoder nets into a full translation pipeline:

In[6]:=

In[7]:=

In[8]:=

In[9]:=

Basic usage

Transcribe speech in different languages:

In[10]:=

Out[10]=

In[11]:=

Out[11]=

In[12]:=

In[13]:=

Out[13]=

Translate a piece of audio from Spanish to English:

In[14]:=

Out[14]=

Whisper can detect the audio language automatically, but the "Language" option can be used to pre-define the language of the audio:

In[15]:=

Out[15]=

Feature extraction

Get a set of English and German audio samples:

In[16]:=

Define a feature extraction using the Whisper encoder:

In[17]:=

Visualize the feature space embedding performed by the audio encoder. Notice that the audio samples from the same class are clustered together:

In[18]:=

Out[18]=

Language identification

Whisper can transcribe and translate audio from 99 languages, with Whisper Large adding support for Cantonese. Retrieve the list of available languages from the label set:

In[19]:=

Out[19]=

Obtain a collection of audio samples featuring speakers of different languages:

In[20]:=

Define a function to detect the language of the audio sample. Whisper determines the language by selecting the most likely language token after the initial pass of the decoder (the following code needs definitions from the "Evaluation function" section):

In[21]:=

Detect the languages:

In[22]:=

Out[22]=

Transcribe and translate the audio samples:

In[23]:=

Out[23]=

Advanced usage

Set the option "IncludeTimestamps" to True to add timestamps at the beginning and end of the audio:

In[24]:=

Out[24]=

In[25]:=

Out[25]=

The option "SuppressSpecialTokens" removes the non-speech tokens. Compare the transcription of the original audio sample with the sample after "SuppressSpecialTokens" is enabled:

In[26]:=

In[27]:=

Out[27]=

In[28]:=

Out[28]=

Transcription and Translation generation

The translation pipeline makes use of two separate transformer nets, encoder and decoder:

In[29]:=

Out[29]=

In[30]:=

Out[30]=

The encoder preprocesses the input audio into a log-Mel spectrogram, capturing the signal's frequency content over time:

In[31]:=

Out[31]=

Get an input audio sample and compute its log-Mel spectrogram:

In[32]:=

In[33]:=

Visualize the log-Mel spectrogram and the audio waveform:

In[34]:=

Out[34]=

The encoder processes the input once, producing a feature matrix of size 1500x768:

In[35]:=

In[36]:=

Out[36]=

The decoding step involves running the decoder multiple times recursively, with each iteration producing a subword token of the translated or transcribed audio. The decoder receives several inputs:

In[37]:=

Out[37]=

• The port "Input1" takes the subword token generated by the previous evaluation of the decoder.

• The port "Index" takes an integer keeping count of how many times the decoder was evaluated (positional encoding).

• The port "Input2" takes the encoded features produced by the encoder. The data fed to this input is the same for every evaluation of the decoder.

• The ports "State1", "State2"... take the self-attention key and value arrays for all the past tokens. Their size grows by one at each evaluation. The default ("Size" -> "Small") decoder has 12 attention blocks, which makes for 24 states: 12 key arrays and 12 value arrays.

The initial prompt for the decoder is a sequence of context tokens that guides Whisper's decoding process by specifying the task to perform and the audio's language. These tokens can be hard-coded to explicitly control the output or left flexible, allowing the model to automatically detect the language and task. Define the initial prompt for transcribing audio in Spanish:

In[38]:=

Retrieve the integer codes of the prompt tokens:

In[39]:=

Out[40]=

Before starting the decoding process, initialize the decoder's inputs:

In[41]:=

In[42]:=

Use the decoder iteratively to transcribe the audio. The recursion keeps going until the EndOfString token is generated or the maximum number of iterations is reached:

In[43]:=

Display the generated tokens:

In[44]:=

Out[44]=

Obtain a readable representation of the tokens by converting the text into UTF8:

In[45]:=

Out[45]=

Change the task type to translate by assigning the third element in the prompt list to "|Translate|":

In[46]:=

Out[47]=

Generate again based on the new prompt:

In[48]:=

Display the generated tokens:

In[49]:=

Out[49]=

Obtain a readable representation of the tokens:

In[50]:=

Out[50]=

Net information

Inspect the number of parameters of all arrays in the net:

In[51]:=

Out[51]=

Obtain the total number of parameters:

In[52]:=

Out[52]=

Obtain the layer type counts:

In[53]:=

Out[53]=

Display the summary graphic:

In[54]:=

Out[54]=