Resource retrieval

Get the pre-trained net:

In[1]:=

Out[1]=

NetModel parameters

This model consists of a family of individual nets, each identified by a specific parameter combination. Inspect the available parameters:

In[2]:=

Out[2]=

Pick a non-default net by specifying the parameters:

In[3]:=

Out[3]=

Pick a non-default uninitialized net:

In[4]:=

Out[4]=

Get the labels:

In[5]:=

Out[5]=

Evaluation function

Write an evaluation function to combine the encoder and decoder nets into a full translation pipeline:

In[6]:=

In[7]:=

In[8]:=

In[9]:=

Basic usage

Transcribe a piece of audio:

In[10]:=

Out[10]=

In[11]:=

Out[11]=

Feature extraction

In[12]:=

Get a set of audio samples for background noise and speech:

In[13]:=

Visualize the feature space embedding performed by the audio encoder. Notice that the human speech samples and the background noise samples belong to different clusters:

In[14]:=

Out[14]=

Advanced usage

Set the option "IncludeTimestamps" to True to add timestamps at the beginning and end of the audio sample:

In[15]:=

Out[15]=

In[16]:=

Out[16]=

Perform transcription with a different "Temperature":

In[17]:=

Out[17]=

The option "SuppressSpecialTokens" removes non-speech tokens. Compare the transcription of the original audio sample with the sample after "SuppressSpecialTokens" is enabled:

In[18]:=

In[19]:=

Out[19]=

In[20]:=

Out[20]=

Whisper can recognize actions or background sounds in an audio sample:

In[21]:=

Out[21]=

In[22]:=

In[23]:=

Out[23]=

Transcription

The transcription pipeline makes use of two separate transformer nets, encoder and decoder:

In[24]:=

Out[24]=

In[25]:=

Out[25]=

The encoder preprocesses input audio into a log-Mel spectrogram, capturing the signal's frequency content over time:

In[26]:=

Out[26]=

Get an input audio sample and compute its log-Mel spectrogram:

In[27]:=

Out[27]=

In[28]:=

Visualize the log-Mel spectrogram and the audio waveform:

In[29]:=

Out[29]=

The encoder processes the input once, producing a feature matrix of size 1500x512:

In[30]:=

In[31]:=

Out[31]=

The decoding step involves running the decoder multiple times recursively, with each iteration producing a subword token of the transcribed audio. The decoder receives several inputs:

In[32]:=

Out[32]=

• The port "Input1" takes the subword token generated by the previous evaluation of the decoder.

• The port "Index" takes an integer keeping count of how many times the decoder was evaluated (positional encoding).

• The port "Input2" takes the encoded features produced by the encoder. The data fed to this input is the same for every evaluation of the decoder.

• The ports "State1", "State2"... take the self-attention key and value arrays for all the past tokens. Their size grows by one at each evaluation. The default ("Size"->"Base") decoder has 12 attention blocks, which makes for 24 states: 12 key arrays and 12 value arrays.

Before starting the decoding process, initialize the decoder's inputs:

In[33]:=

In[34]:=

Use the decoder iteratively to transcribe the audio sample. The recursion keeps going until the EndOfString token is generated or the maximum number of iterations is reached:

In[35]:=

Display the generated tokens:

In[36]:=

Out[36]=

Obtain a readable representation of the tokens:

In[37]:=

Out[38]=

Net information

Inspect the number of parameters of all arrays in the net:

In[39]:=

Out[39]=

Obtain the total number of parameters:

In[40]:=

Out[40]=

Obtain the layer type counts:

In[41]:=

Out[41]=

Display the summary graphic:

In[42]:=

Out[42]=