Recognize Keywords in Speech

Compute the probabilities and locations of a set of words in a recording

Start by downloading and looking at a sample from the "Spoken Digit Commands" training data from the Wolfram Data Repository:

In[1]:=

data=ResourceData["Spoken Digit Commands","TrainingData"];in=RandomChoice@data

Out[1]=

Input

,SpeakerID80b8eab4,Output7

Use the pre-trained speech recognition network from the Wolfram Neural Net Repository to compute the probability that a recording contains a specific word. See the details for this network here. Compute the probabilities of any single letter at all times using the net:

In[2]:=

probabilities=NetModel["Deep Speech 2 Trained on Baidu English Data"][in["Input"],None];MatrixPlotTranspose[probabilities],

opts



Out[2]=

You can use the

CTCLossLayer

to calculate the negative log-likelihood of a specific sequence of characters given the output of the net:

In[3]:=

loss=CTCLossLayer"Input"{"Varying",29},"Target"NetEncoder"Characters",

chars



Out[3]=

CTCLossLayer

Input Ports
Input :	matrix (size: n 1 ×29)
Target :	string
Output Port
Loss :	real



Compute the log-likelihoods of the transcription being one of the digits between 0 and 9:

In[4]:=

choices={"zero","one","two","three","four","five","six","seven","eight","nine"};AssociationMap[loss["Input"probabilities,"Target"#]&,choices]//Sort

Out[4]=

seven0.284794,five26.8032,one31.3448,six33.4508,nine35.5631,three38.0522,zero38.4144,two41.6039,eight42.393,four48.0409

You can do the same operation on a longer audio sample using a sliding window:

In[5]:=

audio=AudioChannelMix[ExampleData[{"Audio","MaleVoice"},"Audio"],1]

Out[5]=

00:00

00:02

Data in Notebook

In[6]:=

NetModel["Deep Speech 2 Trained on Baidu English Data"][%]

Out[6]=

{r,o,h,t, ,t,h,e,n, ,p,e,r,f,e,c,t, ,t,i,m,i,n,g}

Compute the probabilities of any single letter at all times using the net:

In[7]:=

probabilities=NetModel["Deep Speech 2 Trained on Baidu English Data"][audio,None];MatrixPlotTranspose[probabilities],

opts



Out[7]=

Choose the transcription candidates:

In[8]:=

choices={"perfect","then","dog"};

You can partition the probabilities computed by the net to inspect subsets of the signal. The CTC loss can be computed with respect to all of the choices for each partition. This will produce the log-likelihood of a specific choice being the transcription of the specific partition.

BlockMap

is used to apply the function to partitions of the audio signal:

In[9]:=