Wolfram Neural Net Repository

Immediate Computable Access to Neural Net Models

Deep Speech 2 Trained on Baidu English Data (Raw Model)

Transcribe an English speech audio recording

Released in 2017, Baidu Research's Deep Speech 2 model converts speech to text end-to-end from a normalized sound spectrogram to the sequence of characters. It consists of a few convolution layers over both time and frequency, followed by gated recurrent unit (GRU) layers (modified with an additional batch normalization). At evaluation time, the space of possible output sequences is explored by the decoder using a beam search algorithm. The same architecture has also been shown to train successfully on Mandarin Chinese.

Number of layers: 169 | Parameter count: 52,504,416 | Trained size: 211 MB |

Training Set Information

Examples

Resource retrieval

Get the pre-trained network:

In[1]:=
NetModel["Deep Speech 2 Trained on Baidu English Data (Raw Model)"]
Out[1]=

Evaluation function

Write an evaluation function to manage audio preprocessing, net resizing and joining of output characters:

In[2]:=
netevaluate[audio_?AudioQ, beamProperty_ : "Decoding"] := Module[ {net = NetModel[ "Deep Speech 2 Trained on Baidu English Data (Raw Model)"], dec, a = audio, power, encodedInput, resizedNet, netOutput}, If[AudioChannels[a] =!= 1, a = Mean[AudioChannelSeparate[a]]]; If[AudioSampleRate[a] =!= Quantity[16000, "Hertz"], a = AudioResample[a, 16000, Resampling -> {"Hermite", 2}]]; power = AudioMeasurements[a, "Power"]; If[power =!= .01, a = AudioAmplify[a, Sqrt[.01/power]]]; encodedInput = Abs[SpectrogramArray[a, 320, 160, HannWindow][[All, ;; 161]]]^2; dec = NetExtract[net, "Output"]; resizedNet = NetReplacePart[ net, {"Input" -> {First[Dimensions[encodedInput]], 161}, "Output" -> dec}]; netOutput = resizedNet[encodedInput, beamProperty]; Switch[beamProperty, "Decoding", StringJoin[netOutput], "Decodings" | {"TopDecodings", _}, StringJoin /@ netOutput, "NegativeLogLikelihoods" | {"TopNegativeLogLikelihoods", _}, MapAt[StringJoin, netOutput, {All, 1}], None, netOutput, _, $Failed ] ]

Basic usage

Record an audio sample and transcribe it:

In[3]:=
record = AudioCapture[]
Out[3]=
In[4]:=
netevaluate[record]
Out[4]=

Performance evaluation

Models trained with CTC loss have difficulties in producing correct spelling:

In[5]:=
sample = ExampleData[{"Audio", "FemaleVoice"}]
Out[5]=
In[6]:=
netevaluate[sample]
Out[6]=

The standard solution to this problem is to incorporate a language model into the beam search decoding procedure, as done in the original Deep Speech 2 implementation.

Visualization

Write a function to visualize the token probabilities for each time step:

In[7]:=
visualize[audio_] := With[{explikelihood = netevaluate[audio, None]}, MatrixPlot[Transpose@explikelihood, FrameTicks -> {{Transpose@{Range[29], Join[{"'", "<space>"}, Alphabet[], {"<blank>"}]}, None}, {Automatic, Automatic}}, AspectRatio -> 0.8] ]

Inspect the character probabilities of the sample. The <blank> token is the standard blank token used by nets trained with CTC loss:

In[8]:=
visualize[ExampleData[{"Audio", "FemaleVoice"}]]
Out[8]=

Net information

Inspect the sizes of all arrays in the net:

In[9]:=
NetInformation[ NetModel[ "Deep Speech 2 Trained on Baidu English Data (Raw Model)"], "ArraysElementCounts"]
Out[9]=

Obtain the total number of parameters:

In[10]:=
NetInformation[ NetModel[ "Deep Speech 2 Trained on Baidu English Data (Raw Model)"], "ArraysTotalElementCount"]
Out[10]=

Obtain the layer type counts:

In[11]:=
NetInformation[ NetModel[ "Deep Speech 2 Trained on Baidu English Data (Raw Model)"], "LayerTypeCounts"]
Out[11]=

Display the summary graphic:

In[12]:=
NetInformation[ NetModel[ "Deep Speech 2 Trained on Baidu English Data (Raw Model)"], "FullSummaryGraphic"]
Out[12]=

Export to MXNet

Export the net into a format that can be opened in MXNet:

In[13]:=
jsonPath = Export[FileNameJoin[{$TemporaryDirectory, "net.json"}], NetModel["Deep Speech 2 Trained on Baidu English Data (Raw Model)"], "MXNet"]
Out[13]=

Export also creates a net.params file containing parameters:

In[14]:=
paramPath = FileNameJoin[{DirectoryName[jsonPath], "net.params"}]
Out[14]=

Get the size of the parameter file:

In[15]:=
FileByteCount[paramPath]
Out[15]=

The size is similar to the byte count of the resource object:

In[16]:=
ResourceObject[ "Deep Speech 2 Trained on Baidu English Data (Raw Model)"]["ByteCount"]
Out[16]=

Requirements

Wolfram Language 11.3 (March 2018) or above

Resource History

  • Date Created: 27 March 2018

Reference