Resource retrieval

Get the pre-trained net:

In[1]:=

Out[1]=

NetModel parameters

This model consists of a family of individual nets, each identified by a specific parameter combination. Inspect the available parameters:

In[2]:=

Out[2]=

Pick a non-default net by specifying the parameters:

In[3]:=

Out[3]=

Pick a non-default uninitialized net:

In[4]:=

Out[4]=

Get the labels:

In[5]:=

Out[5]=

Evaluation function

Write an evaluation function to combine the encoder and decoder nets into a full transcription pipeline:

In[6]:=

In[7]:=

In[8]:=

In[9]:=

Basic usage

Transcribe speech in English:

In[10]:=

Out[10]=

In[11]:=

Out[11]=

Transcribe speech in Spanish:

In[12]:=

In[13]:=

Out[14]=

Whisper can detect the audio language sample automatically, but the "Language" option can be used to pre-define the language of the audio sample. Transcribe speech in Japanese:

In[15]:=

In[16]:=

Out[16]=

Set the option "IncludeTimestamps" to True to add timestamps at the beginning and end of the audio sample:

In[17]:=

Out[17]=

Feature extraction

Get a set of audio samples with human speech and background noise:

In[18]:=

Define a feature extraction using the Whisper encoder:

In[19]:=

Visualize the feature space embedding performed by the audio encoder. Notice that the audio samples from the same class are clustered together:

In[20]:=

Out[20]=

Get a set of English and German audio samples:

In[21]:=

Visualize the feature space embedding and observe how the English and German audio samples are clustered together:

In[22]:=

Out[22]=

Language identification

Whisper can transcribe from one hundred different languages. Retrieve the list of available languages from the label set:

In[23]:=

Out[23]=

Obtain a collection of audio samples featuring speakers of different languages:

In[24]:=

Define a function to detect the language of the audio sample. Whisper determines the language by selecting the most likely language token after the initial pass of the decoder (the following code needs definitions from the "Evaluation function" section):

In[25]:=

Detect the languages:

In[26]:=

Out[26]=

Transcribe the audio samples:

In[27]:=

Out[27]=

Transcription generation

The transcription pipeline makes use of two separate transformer nets, encoder and decoder:

In[28]:=

Out[28]=

In[29]:=

Out[29]=

The encoder preprocesses the input audio into a log-Mel spectrogram, capturing the signal's frequency content over time:

In[30]:=

Out[30]=

Get an input audio sample and compute its log-Mel spectrogram:

In[31]:=

In[32]:=

Visualize the log-Mel spectrogram and the audio waveform:

In[33]:=

Out[33]=

The encoder processes the input once, producing a feature matrix of size 1500x1280:

In[34]:=

In[35]:=

Out[35]=

The decoding step involves running the decoder multiple times recursively, with each iteration producing a subword token of the transcribed audio. The decoder receives several inputs:

In[36]:=

Out[36]=

• The port "Input1" takes the subword token generated by the previous evaluation of the decoder.

• The port "Index" takes an integer keeping count of how many times the decoder was evaluated (positional encoding).

• The port "Input2" takes the encoded features produced by the encoder. The data fed to this input is the same for every evaluation of the decoder.

• The ports "State1", "State2"... take the self-attention key and value arrays for all the past tokens. Their size grows by one at each evaluation. The decoder has four attention blocks, which makes for eight states: four key arrays and four value arrays.

The initial prompt is a sequence of context tokens that guides Whisper's decoding process by specifying the task to perform and the audio's language. These tokens can be hard-coded to explicitly control the output or left flexible, allowing the model to automatically detect the language and task. Define the initial prompt for transcribing audio in Spanish:

In[37]:=

Retrieve the integer codes of the prompt tokens:

In[38]:=

Out[39]=

Before starting the decoding process, initialize the decoder's inputs:

In[40]:=

In[41]:=

Use the decoder iteratively to transcribe the audio. The recursion keeps going until the EndOfString token is generated or the maximum number of iterations is reached:

In[42]:=

Display the generated tokens:

In[43]:=

Out[43]=

Obtain a readable representation of the tokens by converting the text into UTF8:

In[44]:=

Out[44]=

Net information

Inspect the number of parameters of all arrays in the net:

In[45]:=

Out[45]=

Obtain the total number of parameters:

In[46]:=

Out[46]=

Obtain the layer type counts:

In[47]:=

Out[47]=

Display the summary graphic:

In[48]:=

Out[48]=