Audio Analysis with Neural Networks

Audio Encoding

The fundamental tool to transform (RefLink[Audio, paclet:ref/Audio]) objects (or audio files) into a format appropriate for neural nets is the (RefLink[NetEncoder, paclet:ref/NetEncoder]). The Wolfram Language natively provides several audio encoders that are based on different kinds of feature computations. These encoders all leverage a low-level, parallel implementation that allows for a very fast computation.

Encoder Types

All the encoders share the same preprocessing steps. The first is the extraction of the appropriate part of the signal, followed by downmixing to a single channel and resampling to a uniform frequency.

The available audio encoders are:

Audio encoders.

The (StringTypeLink["Audio", paclet:ref/netencoder/Audio]) encoder simply extracts the waveform of the signal. This preserves all information, but the dimensions of the result are not ideal for neural nets.

The (StringTypeLink["AudioSTFT", paclet:ref/netencoder/Audio]) encoder partitions the signal and computes a (RefLink[Fourier, paclet:ref/Fourier]) transform on each partition (the whole operation is called short-time Fourier transform, or STFT). This provides both time and frequency information, and since the Fourier transform is invertible, all the information in the original signal is preserved.

The (StringTypeLink["AudioSpectrogram", paclet:ref/netencoder/Audio]) encoder computes the squared magnitude of the STFT, and discards the redundant part. This reduces the dimensionality of the feature, but the phase information is lost. It is still possible to compute an approximate reconstruction of the original signal (see (RefLink[InverseSpectrogram, paclet:ref/InverseSpectrogram]) for details).

The (StringTypeLink["AudioMelSpectrogram", paclet:ref/netencoder/Audio]) encoder applies a filterbank to the spectrogram. The center frequencies of the filterbank are spaced linearly on the mel scale, a nonlinear frequency scale that mimics the human perception of pitch. This reduces the dimensionality even further.

The (StringTypeLink["AudioMFCC", paclet:ref/netencoder/Audio]) encoder computes the (RefLink[FourierDCT, paclet:ref/FourierDCT]) of the logarithm of the mel-spectrogram and discards some of the higher-order coefficients. This achieves a very high dimensionality reduction while preserving a lot of the important information, especially for speech signals.

Plot the result of the (StringTypeLink["AudioMelSpectrogram", paclet:ref/netencoder/AudioMelSpectrogram]) encoder:

enc=NetEncoder["AudioMelSpectrogram"]

enc[]//Short

MatrixPlot[Log[Transpose[%]],]

Data Augmentation

All audio encoders also share the "Augmentation" parameter. This allows them to perform data augmentation on the input, before the features (such as the spectrogram or the MFCC) are computed.

Data augmentation can be very useful when dealing with limited or reduced-size datasets, and to make a net more robust against artificial/irrelevant trends in the training data. As an example, if you were to classify recordings of cats and dogs, and in the training data all the dogs were recorded with a noisy microphone, the network might end up recognizing the noise rather than the dog.

Another convenient usage of data augmentation for audio is extracting segments from the recordings in the training data.

The available augmentations are:

All of the augmentations can be randomized.

Apply two augmentations at the same time:

enc=NetEncoder[{"Audio","Augmentation"->{"Noise"->{{.1,1},},"Convolution"->{{.1,.5},}}}]

Audio[enc[]//Transpose,SampleRate->16000]

Audio Classification

Train a classification network on the "Spoken Digit Commands" dataset.

The dataset consists of 10K recordings of spoken digits from 0 to 9, by a collection of different speakers. The speakers of the training and testing portions of the dataset do not overlap.

Gather the data:

ro=ResourceObject["Spoken Digit Commands"];
trainingSet=ResourceData[ro,"TrainingDataset"];
testingSet=ResourceData[ro,"TestDataset"];

RandomSample[trainingSet,3]

Use the "AudioMFCC" encoder, since it provides significant dimensionality reduction while preserving a lot of the information present in speech signals.

Create the encoder and decoder:

enc=NetEncoder[{"AudioMFCC","WindowSize"->(Quantity[40, "ms", milliseconds, "Milliseconds"]),"Offset"->(Quantity[20, "ms", milliseconds, "Milliseconds"]),"NumberOfCoefficients"->20}]

dec=NetDecoder[{"Class",Range[0,9]}]

(RefLink[ConvolutionLayer, paclet:ref/ConvolutionLayer]) supports variable-length inputs. This is achieved by setting the "Interleaving" option to (RefLink[True, paclet:ref/True]). Then a very simple convolutional net can be built based on the "LeNet" architecture, which is widely used in image processing. Some adjustments in the final layers need to be made to accommodate the variable-length nature of audio data.

Create the convolutional net, train it on the dataset and measure its performance:

convNet=NetChain[{ConvolutionLayer[32,5,"Interleaving"->True],Ramp,ConvolutionLayer[64,5,"Interleaving"->True],Ramp,ConvolutionLayer[10,1,"Interleaving"->True],AggregationLayer[Max,1],SoftmaxLayer[]},"Input"->enc,"Output"->dec]

trainedConvNet=NetTrain[convNet,trainingSet,ValidationSet->Scaled[.05]];

ClassifierMeasurements[trainedConvNet,#Input->#Output&/@Normal[testingSet],"Report"]

Another approach is to create a fully recurrent network. The net is based on a stack of (RefLink[GatedRecurrentLayer, paclet:ref/GatedRecurrentLayer]), followed by a simple classification section. To add some regularization, dropout at the input of the recurrent layer is used.

Create the recurrent net, train it on the dataset and measure its performance:

recurrentNetDropout=NetChain[{GatedRecurrentLayer[64,"Dropout"->{"VariationalInput"->.2}],GatedRecurrentLayer[64,"Dropout"->{"VariationalInput"->.2}],SequenceLastLayer[],LinearLayer[],SoftmaxLayer[]},"Input"->enc,"Output"->dec]

traindRecurrentNetDropout=NetTrain[recurrentNetDropout,trainingSet,ValidationSet->Scaled[.05]]

ClassifierMeasurements[traindRecurrentNetDropout,#Input->#Output&/@Normal[testingSet],"Report"]

By removing the last classification layers, it is possible to obtain a feature extractor for audio signals. This feature extractor can be relevant for features that were important for the digit classification task; in other words, the learned embedding will contain information on which digit was spoken, but will disregard any information regarding the speaker identity.

Plot the testing dataset using the embedding learned by the network:

FeatureSpacePlot[Style[#Input,ColorData[97][#Output]]->#Output&/@testingSet,FeatureExtractor->NetTake[traindRecurrentNetDropout,3],LabelingFunction->Callout]

Audio Event Detection

In some cases, the goal is to train a network to locate sound events in a recording, but the data is "weakly labeled", where the labels only state if a certain event was present in a recording, but not where. Despite the limitation of the data, it is possible to obtain good results in sound event localization through training on weakly labeled data.

Use the "Audio Cats and Dogs" dataset from the Wolfram Data Repository, a collection of recordings of cats and dogs.

Retrieve the dataset:

ro=ResourceObject["Audio Cats and Dogs"];
data=ResourceData[ro];
classes=Normal[data[Union,"Label"]];
RandomSample[data,2]

The duration of the signals varies between 1 and 18 seconds.

Divide the data into training and testing sets and plot a histogram of the duration of the signals:

data=Map[#Audio->#Label&,Normal[data]];
{trainingData,testingData}=TakeDrop[RandomSample[data],Round[.8Length[data]]];

Histogram[Duration/@data[[All,1]]]

Use the (StringTypeLink["AudioMelSpectrogram", paclet:ref/netencoder/AudioMelSpectrogram]) encoder to feed the audio signal into the network. Since the amount of data is relatively small, data augmentation can be done to make the training more effective.

Create the encoder and decoder:

enc=NetEncoder[{"AudioMelSpectrogram","WindowSize"->(Quantity[40, "ms", milliseconds, "Milliseconds"]),"Offset"->(Quantity[20, "ms", milliseconds, "Milliseconds"]),"NumberOfFilters"->64,"Augmentation"->{"TimeShift"->Scaled[{-.5,0}],"VTLP"->{0.8,1.2}}}]

dec=NetDecoder[{"Class",Union[data[[All,2]]]}]

The net will be based on a stack of recurrent layers ((RefLink[GatedRecurrentLayer, paclet:ref/GatedRecurrentLayer])), and an (RefLink[AggregationLayer, paclet:ref/AggregationLayer]) to pool the result in the time dimension. This allows the net to output a single classification result instead of a sequence.

Create the net and train it on the dataset:

net=
NetChain[{ElementwiseLayer[Log[#+.01]&],GatedRecurrentLayer[64,"Dropout"->{"VariationalInput"->.2}],GatedRecurrentLayer[64,"Dropout"->{"VariationalInput"->.2}],GatedRecurrentLayer[2],AggregationLayer[Max,1],SoftmaxLayer[]},
"Input"->enc,"Output"->NetDecoder[{"Class",classes}]]

trainedNet=NetTrain[net,trainingData,ValidationSet->Scaled[.08],MaxTrainingRounds->40]

To use the net in a reproducible fashion, the encoder with the augmentations needs to be replaced with one without them.

Measure the performance of the net:

finalNet=NetReplacePart[trainedNet,"Input"->NetEncoder[{"AudioMelSpectrogram","WindowSize"->(Quantity[40, "ms", milliseconds, "Milliseconds"]),"Offset"->(Quantity[20, "ms", milliseconds, "Milliseconds"]),"NumberOfFilters"->64}]]

ClassifierMeasurements[finalNet,testingData,"Report"]

By removing the (RefLink[AggregationLayer, paclet:ref/AggregationLayer]) and reattaching the (RefLink[SoftmaxLayer, paclet:ref/SoftmaxLayer]) to the chopped net, a network that returns a sequence of class probabilities instead of a single classification result is obtained.

Create the time-resolved net and define a function to easily get its results:

timeResolvedNet=NetAppend[NetTake[finalNet,4],SoftmaxLayer[]]

generateTimeSeries[in_Audio]:=Association[MapThread[#2->TimeSeries[#1,{0,QuantityMagnitude[Duration[in],"Seconds"]}]&,{Transpose[Normal[timeResolvedNet[in]]],classes}]]

Test this time-resolved net on a signal that contains both cat and dog noises.

Test the net on a signal that was not in the training set:

aud=;
aud//AudioAnnotationLookup//Dataset

getIntervals[in_,thr_:.51]:=Map[If[#[[1,2]]===1,{#[[1,1]],#[[-1,1]]},Nothing]&,SplitBy[MapAt[If[#>thr,1,0]&,Normal[in],{All,2}],#[[2]]&]]
getRectangles[in_]:=Map[Rectangle[{#[[1]],-1},{#[[2]],1}]&,in]

intervals=getIntervals/@generateTimeSeries[aud];

Legended[AudioPlot[aud,ImageSize->Medium,Epilog->{,getRectangles[intervals["cat"]],,getRectangles[intervals["dog"]]},AnnotationRules->"animal",AspectRatio->1/2],SwatchLegend[{,},{"cat","dog"}]]

Audio Embeddings

An embedding can be learned as side effect of training on a classification task. It is possible to use different training tasks to make the learning of a meaningful embedding the objective of the training.

Siamese Networks with Contrastive Loss

The first strategy to learn an embedding is training a siamese network using the contrastive loss. This involves feeding two inputs to the exact same network, computing the distance between the two outputs and feeding it to the contrastive loss. If the two inputs belong to the same class, the distance will be minimized; if they do not, it will be maximized.

This will again use the "Spoken Digit Commands" dataset. This time around, ignore which digit was spoken and only pay attention to the speaker.

Gather the data and group it by the speaker ID:

ro=ResourceObject["Spoken Digit Commands"];
trainingSet=Normal@ResourceData[ro,"TrainingDataset"];
testingSet=Normal@ResourceData[ro,"TestDataset"];

groupedTrain=GroupBy[trainingSet,#SpeakerID&];
groupedTrain=Select[groupedTrain,Length[#]≥2&];
Keys[groupedTrain]//Short

Create a set of pairs of of example speakers. Ideally, it should have as many positive examples (recordings from the same speakers) as negative examples (recordings from different speakers).

Create the training pairs:

embeddingData=Flatten@Table[With[{speaker=RandomChoice[Keys[groupedTrain]]},{<|"Input1"->RandomChoice[groupedTrain[speaker]]["Input"],"Input2"->RandomChoice[RandomChoice[KeyDrop[groupedTrain,speaker]]]["Input"],"Target"->True|>,
<|"Input1"->#[[1,"Input"]],"Input2"->#[[2,"Input"]],"Target"->False|>&@RandomSample[groupedTrain[speaker],2]}],30000];

This example will use a very simple recurrent network to produce the embedding. It will also define a net to train the embedding. This will need to feed the two inputs to the same network and compute the distance of the resulting embeddings. Use (RefLink[NetInsertSharedArrays, paclet:ref/NetInsertSharedArrays]) to ensure that the two inputs are processed by exactly the same network.

Create the net that will produce the embedding and the net to compute the contrastive loss:

embeddingNet=NetChain[{GatedRecurrentLayer[64,"Dropout"->{"VariationalInput"->.2}],GatedRecurrentLayer[64,"Dropout"->{"VariationalInput"->.2}],SequenceLastLayer[]},"Input"->NetEncoder[{"AudioMFCC","WindowSize"->(Quantity[40, "ms", milliseconds, "Milliseconds"]),"Offset"->(Quantity[20, "ms", milliseconds, "Milliseconds"]),"NumberOfCoefficients"->20}]]

pairEmbeddingNet=With[{netWithSharedArrays = NetInsertSharedArrays[embeddingNet]},	NetGraph[<|"net1" -> netWithSharedArrays,"net2" -> netWithSharedArrays,"distance" -> MeanSquaredLossLayer[],"loss"->ContrastiveLossLayer[]|>,{NetPort["Input1"] -> "net1" -> NetPort["distance", "Input"],NetPort["Input2"] -> "net2" -> NetPort["distance", "Target"],"distance" -> NetPort["loss","Input"]}]]

Once the network is trained, extract one of the equivalent subnets that computes the embedding.

Train the pair-embedding net, and extract the embedding net from the trained result:

trainedEmbedding=NetTrain[pairEmbeddingNet,embeddingData,ValidationSet->Scaled[.05]]

embedding=NetTake[trainedEmbedding,"net1"]

Now use the embedding to visualize collections of signals.

Visualize the test dataset in the computed embedding space:

groupedTest=SortBy[GroupBy[testingSet,#SpeakerID&],Minus@*Length];
color=AssociationThread[Keys[groupedTest]->(ColorData[97]/@Range[Length[groupedTest]])];

FeatureSpacePlot[Flatten@Values@Map[Style[#Input,color[#SpeakerID]]->#SpeakerID&,groupedTest[[All,;;UpTo[20]]],{2}],FeatureExtractor->embedding,LabelingFunction->Callout]

Or compare different recordings by measuring the distance between their embeddings.

Plot the distance matrix between different examples in the test dataset:

sample=RandomSample[groupedTest,10];

plotDistanceMatrix[data_,embedding_]:=Module[
{lengths,bounds,ticks,matrix},
lengths=Length/@sample;
bounds=Accumulate[Values[lengths]];
ticks=MapThread[{Mean[#1],#2}&,{Partition[Prepend[bounds,1],2,1],Keys[lengths]}];
matrix=DistanceMatrix[Flatten[Values[data]][[All,"Input"]],FeatureExtractor->embedding];
MatrixPlot[matrix,FrameTicks->{{ticks,None},{MapAt[Rotate[#,Pi/2]&,ticks,{All,2}],None}},Mesh->{bounds,bounds}
]
]

plotDistanceMatrix[sample,embedding]

Define a (RefLink[NearestFunction, paclet:ref/NearestFunction]) using the learned embedding, and find the closest and furthest signals from an example:

fn=FeatureNearest[testingSet[[All,"Input"]],FeatureExtractor->embedding]

aud=RandomChoice[trainingSet]

AudioNormalize/@fn[aud["Input"],3]
AudioNormalize/@fn[aud["Input"],All][[-3;;]]

Pre-trained Audio Feature Extractors

An alternative to learning an embedding from scratch is to leverage pre-trained audio analysis networks. The (WebLink[Wolfram Neural Net Repository, https://resources.wolframcloud.com/NeuralNetRepository/]) is an excellent source.

(RefLink[AudioIdentify, paclet:ref/AudioIdentify]) uses a deep neural net as a back end to perform audio classification. The network was trained on the (WebLink[AudioSet, https://research.google.com/audioset/]) dataset, where each audio signal is annotated with the sound classes/sources that are present in the recording. The labels are organized in an ontology of about 600 classes that span a very wide domain of sound types or sources, from musical instruments and music types to animal, mechanical and human sounds.

Import the AudioIdentify net and apply it on an (RefLink[Audio, paclet:ref/Audio]) object:

net=NetModel["Wolfram AudioIdentify V1 Trained On AudioSet Data","Size"->"Small"]

net[,"TopProbabilities"]

In the AudioIdentify net, the signal is divided into fixed-size chunks, and the main net is applied to the mel-spectrogram of each of those chunks. Use (RefLink[NetExtract, paclet:ref/NetExtract]) to get it. Similarly to the architectures presented in (WebLink["CNN Architectures for Large-Scale Audio Classification", https://arxiv.org/abs/1609.09430]), the main net has a CNN architecture (based on (WebLink[MobileNet v2, https://arxiv.org/abs/1801.04381])).

Extract the net at the core of the (RefLink[NetMapOperator, paclet:ref/NetMapOperator]):

net=NetModel["Wolfram AudioIdentify V1 Trained On AudioSet Data","Size"->"Small"]

mainNet=NetExtract[net,{1,"Net"}]

The network used in (RefLink[AudioIdentify, paclet:ref/AudioIdentify]) can be used not only for recognizing sounds, but also to extract features from a recording. This allows any signal to be embedded in a semantically meaningful space, where similarities and distances can be computed.

The last few layers that are in charge of the classification task can be removed, and the resulting network can be reinserted into the original (RefLink[NetChain, paclet:ref/NetChain]). This net will produced a fixed-size, semantically meaningful vector for each audio input. It can be used as a feature extractor for all the the high-level machine learning functions in the system, or as a starting point to train a new neural net.

Create a feature extractor net and use it in (RefLink[FeatureSpacePlot, paclet:ref/FeatureSpacePlot]):

featureExtractor=NetChain[{NetMapOperator[NetDelete[mainNet,{158,159,160}]],AggregationLayer[Max,1],FlattenLayer[]},"Input"->net[["Input"]]]

FeatureSpacePlot[ExampleData[#]->#[[2]]&/@ExampleData["Audio"],FeatureExtractor->featureExtractor,LabelingFunction->Callout]

Another alternative is the "VGGish Feature Extractor Trained on YouTube Data" model, a structurally similar net that was trained by Google on data from YouTube specifically for audio feature extraction.

Use the VGGish network as a feature extractor in (RefLink[FeatureSpacePlot, paclet:ref/FeatureSpacePlot]):

vggish=NetModel["VGGish Feature Extractor Trained on YouTube Data"]

FeatureSpacePlot[ExampleData[#]->#[[2]]&/@ExampleData["Audio"],FeatureExtractor->vggish,LabelingFunction->Callout]

Transfer Learning for Audio

Sometimes the amount of data available to train a network is insufficient for the task at hand. Transfer learning is a possible solution to this problem. Instead of training a network from scratch, it is possible to use as a starting point a net that has already been trained on a different but related task.

Begin by downloading the (WebLink[ESC-50 dataset, https://github.com/karoldvl/ESC-50]).

Download and parse the ESC-50 dataset:

archive=URLDownload["https://github.com/karoldvl/ESC-50/archive/master.zip"];
dir=FileNameJoin[{$TemporaryDirectory,"ESC50"}];ExtractArchive[archive,dir];

rawMetadata=Import[FileNameJoin[{dir,"ESC-50-master","meta","esc50.csv"}]];
metaData=Map[Append[AssociationThread[rawMetadata[[1]]->MapAt[File@FileNameJoin[{dir,"ESC-50-master","audio",#}]&,#,1]],"features"->None]&,Rest[rawMetadata]];

RandomChoice[metaData]//Dataset

classes=DeleteDuplicates[metaData[[All,"category"]]];
classes//Short

Use the network from (RefLink[AudioIdentify, paclet:ref/AudioIdentify]) as a starting point.

Construct the feature extraction network:

net=NetModel["Wolfram AudioIdentify V1 Trained On AudioSet Data","Size"->"Large"];
mainNet=NetExtract[net,{1,"Net"}]

featureExtractor=NetChain[{NetMapOperator[NetDelete[mainNet,{158,159,160}]],ReshapeLayer[{Inherited,Automatic}]},"Input"->net[["Input"]]]

Instead of retraining the full net and specifying a (RefLink[LearningRateMultipliers, paclet:ref/LearningRateMultipliers]) option in (RefLink[NetTrain, paclet:ref/NetTrain]) to train only the classification layers, precompute the results of the feature extractor net and train the classifier. This avoids redundant evaluation of the full net during training.

It is also possible to divide the dataset into the folds defined by its creator.

Precompute the features and divide the dataset according to the original folds:

metaData[[All,"features"]]=featureExtractor[metaData[[All,"filename"]]];

folds=Table[<|"Train"->(#features->#category&/@Select[metaData,#fold=!=i&]),"Test"->(#features->#category&/@Select[metaData,#fold===i&])|>,{i,5}];

Construct a simple recurrent classifier network that will be attached to the feature extractor and train it on each of the folds.

Define a simple recurrent classifier and train it on each of the folds:

classifier=NetChain[{GatedRecurrentLayer[128,"Dropout"->{"VariationalInput"->.1}],AggregationLayer[Mean,1],LinearLayer[],SoftmaxLayer[]},"Input"->{"Varying",1664},"Output"->NetDecoder[{"Class",classes}]]

trainedClassifiers=Table[
NetTrain[classifier,data["Train"],ValidationSet->Scaled[.05],MaxTrainingRounds->20],
{data,folds}];

After training, it is easy to measure the accuracy of the classifiers on each of the folds and average them to obtain a cross-validated result.

Measure the performance on each fold and compute their average:

MapThread[NetMeasurements[#1,#2,"Accuracy"]&,{trainedClassifiers,folds[[All,"Test"]]}]

Mean[%]

As a final test, construct a net that joins the original feature extractor and an ensemble average of the trained classifiers, and run it on a signal that was not in the original dataset.

Test the final net on an unrelated example:

finalNet=NetGraph[Join[trainedClassifiers,{TotalLayer[],ElementwiseLayer[#/5&],featureExtractor}],{NetPort["Input"]->8->Range[5]->6->7},"Output"->NetDecoder[{"Class",classes}]]

finalNet[,{"TopProbabilities",3}]//Reverse