Audio Processing

The Wolfram Language provides built-in support for both programmatic and interactive audio processing, fully integrated with other powerful mathematical and algorithmic capabilities. You can process audio objects by applying linear and nonlinear filters, add effects, and analyze them using audio-specific functions or by exploiting the extensive integration with the rest of the Wolfram Language.

Filtering

Audio signals can be used as input to many signal processing functions.

Some of the filters that can be directly applied to audio objects.

Many of the filtering functions present in the Wolfram Language can be immediately used on audio objects. In many cases, it is possible to specify the cutoff frequency as a frequency (RefLink[Quantity, paclet:ref/Quantity]).

FIR filters applied to an audio object:

a=ExampleData[{"Audio","Apollo11ReturnSafely"},"Audio"]

lp=LowpassFilter[a,(Quantity[800, "Hz", hertz, "Hertz"]),101]

hp=HighpassFilter[a,(Quantity[1600, "Hz", hertz, "Hertz"]),101]

bs=BandstopFilter[a,{(Quantity[800, "Hz", hertz, "Hertz"]),(Quantity[1600, "Hz", hertz, "Hertz"])},91]

Plot the periodogram of the original and processed signals:

Periodogram[{a,bs,lp,hp},2000,PlotRange->{{0,3000},Automatic},Frame->True,ImageSize->Medium,PlotLegends->{"Original","Bandstop","Lowpass","Highpass"}]

Use (RefLink[WienerFilter, paclet:ref/WienerFilter]) to denoise a recording:

a=ExampleData[{"Audio","Apollo11ReturnSafely"},"Audio"]

WienerFilter[a,25]

Spectrogram[#,ImageSize->Small,FrameTicks->None]&/@{a,%}

Discrete-time transfer function models can be used to filter an audio object using (RefLink[RecurrenceFilter, paclet:ref/RecurrenceFilter]).

Some of the available filter models and additional utilities.

The simplest way to use one of the analog (continuous-time) filter models is to discretize the transfer function using (RefLink[ToDiscreteTimeModel, paclet:ref/ToDiscreteTimeModel]) and apply the result to an audio object using (RefLink[RecurrenceFilter, paclet:ref/RecurrenceFilter]).

Apply a resonant biquadratic filter to an audio object:

filterModel=BiquadraticFilterModel[{ω,q}]

discreteFilter=ToDiscreteTimeModel[filterModel,1/22050,z,Method->{"BilinearTransform","CriticalFrequency"->ω}]

a=;

RecurrenceFilter[discreteFilter/.{ω->1000 2 Pi,q->7},a]

A discrete transfer function can be created using (RefLink[TransferFunctionModel, paclet:ref/TransferFunctionModel]) and applied to an audio object with (RefLink[RecurrenceFilter, paclet:ref/RecurrenceFilter]).

Define a comb filter using (RefLink[TransferFunctionModel, paclet:ref/TransferFunctionModel]) and apply it to an audio object:

combFilterModel[f_,α_,sr_:22050]:=TransferFunctionModel[{{1/( 1-α (z^(-Round[sr/f])))}},z,SamplingPeriod->(1/sr)]

combFilterModel[150,α]

RecurrenceFilter[combFilterModel[150,-.9],a]

Effects and Manipulation

An audio object can be modified and manipulated using built-in or user-defined functions.

Some common audio effects.

Use pitch shifting and time stretching to independently modify pitch and duration of an audio signal.

Slow down an audio object:

a=ExampleData[{"Audio","MaleVoice"},"Audio"]

AudioTimeStretch[a,1.5]

Shift the pitch of an audio object up:

AudioPitchShift[a,(Quantity[2, "semitones", semitones, IndependentUnit["semitones"]])]

Delay and reverberation effects can be used to immerse a recording in a virtual environment or to produce special effects.

Apply a delay or reverb effect:

AudioDelay[a,.7,.8,.4,PaddingSize->2]

AudioTrim[AudioReverb[a,ExampleData[{"Audio","IRStMarysChurch"}]],10]

Perform Karplus–Strong synthesis by adding a short delay with a high feedback value to a burst of noise. This will simulate the sound of a vibrating string:

freq=60;
feedback=0.99;
AudioDelay[AudioGenerator["Pink",.01],1/freq,feedback,1,PaddingSize->5,Method->{"LowpassCutoff"->(Quantity[8000, "Hz", hertz, "Hertz"])}]

Downmixing and upmixing to an arbitrary number of channels can be achieved using (RefLink[AudioChannelMix, paclet:ref/AudioChannelMix]).

Downmix and upmix a multichannel audio object:

a//AudioChannels

AudioChannelMix[a,"Mono"]//AudioChannels

AudioChannelMix[a,3]//AudioChannels

It is possible to alter a recording by taking advantage of performing arithmetic operations on the (RefLink[Audio, paclet:ref/Audio]) object. All Wolfram Language operators and functions with attributes (RefLink[NumericFunction, paclet:ref/NumericFunction]) or (RefLink[Listable, paclet:ref/Listable]) are overloaded to work with audio objects.

Apply a smooth distortion to an audio object using the (RefLink[Tanh, paclet:ref/Tanh]) function:

a=AudioNormalize[Import["ExampleData/Rule30.wav"]]

Tanh[5a]

Use the (RefLink[ChebyshevT, paclet:ref/ChebyshevT]) function to obtain a "waveshaper" effect:

ChebyshevT[10,a]

Multiply an audio object with a sine wave to get a "ring modulator" effect:

a×

Analysis

Analysis of the Whole Signal

Properties can be computed on a recording both on a global and a local scale.

Some of the functionality to compute global measurements.

Both time domain and frequency domain properties can be measured with (RefLink[AudioMeasurements, paclet:ref/AudioMeasurements]). The properties are computed on the average sample values over the channels of the audio object.

Compute time domain properties of a recording:

AudioMeasurements[ExampleData[{"Audio","Drums"}],{"MinMax","Mean","StandardDeviation","RMSAmplitude","ZeroCrossings"},"Dataset"]

Compute frequency domain properties of a recording:

AudioMeasurements[ExampleData[{"Audio","Drums"}],{"SpectralCentroid","SpectralFlatness","SpectralRollOff","SpectralSpread"},"Dataset"]

Compute other measurements by directly applying built-in statistical functions.

Compute moment and entropy of an audio object:

a=AudioGenerator["Sin"];

Moment[a,2]

Entropy[a]

Unlike (RefLink[AudioMeasurements, paclet:ref/AudioMeasurements]), overloaded functions are applied to the flattened version of the data. If the input is a multichannel audio object, the sample values from all channels will be flattened in a single array.

Compute statistical properties of a recording:

AssociationThread[{"MinMax","Mean","StandardDeviation","RMSAmplitude","ZeroCrossings"} ->
Through[{MinMax,Mean,StandardDeviation,Sqrt@Mean[#^2]&,Total@CrossingDetect[#,CornerNeighbors->None]&}@ExampleData[{"Audio","Drums"}]]
]//Dataset

Analysis of the Partitioned Signal

In addition to global properties of audio objects, it is also possible to compute measurements locally.

Some of the functionality to compute local measurements..

In (RefLink[AudioLocalMeasurements, paclet:ref/AudioLocalMeasurements]), properties are computed locally. The signal is partitioned according to the (RefLink[PartitionGranularity, paclet:ref/PartitionGranularity]) specification, and the requested property is computed on each partition. The result is returned as a (RefLink[TimeSeries, paclet:ref/TimeSeries]) whose timestamps correspond to the center of each partition.

Compute the RMS amplitude on 40 ms partitions with an offset of 1 ms:

a=;res=AudioLocalMeasurements[a,"RMSAmplitude",PartitionGranularity->{(Quantity[40, "ms", milliseconds, "Milliseconds"]),(Quantity[1, "ms", milliseconds, "Milliseconds"])}];
Show[AudioPlot[a],ListLinePlot[res,PlotStyle->Red]]

The result of (RefLink[AudioLocalMeasurements, paclet:ref/AudioLocalMeasurements]) or (RefLink[AudioMeasurements, paclet:ref/AudioMeasurements]) can be used as an input for other functionality in the Wolfram Language.

Use the "SpectralCentroid" and "SpectralSpread" measurements to find clusters of similar audio objects in a list:

samples={,,,,,,,,};
props=AudioMeasurements[#,{"SpectralCentroid","SpectralSpread"},"List"]&/@samples;
clusters=FindClusters[props->props]

ListPlot[Partition[props,1],PlotMarkers->samples,ImageSize->300,Prolog->({Opacity[.3,],Disk[Mean@#,400]&/@clusters}),PlotRange->{{00,4000},{0,4000}},AspectRatio->1]

Use the MFCC measurement as a feature to compute the distance between various elements of the (RefLink[ExampleData, paclet:ref/ExampleData])["Audio"] collection:

list=Select[ExampleData["Audio"],ExampleData[#,"Duration"]<10&];
a=ConformAudio[AudioNormalize@AudioChannelMix[#,1]&/@ExampleData[#,"Audio"]&/@list,SampleRate->11025];

mfcc=AudioLocalMeasurements[#,"MFCC",PartitionGranularity->{.05,.01}]["Values"]&/@a;

ticks=Thread[{Range[Length@list],Text/@list[[All,2]]}];MatrixPlot[DistanceMatrix[mfcc],ImageSize->Medium,FrameTicks->{ticks,Apply[Rotate[#,Pi/2]&,ticks,{2}]}]

Using (RefLink[AudioIntervals, paclet:ref/AudioIntervals]) allows you to extract intervals on which a user-defined criterion is satisfied.

Locate the non-voiced sections of a recording:

a=ExampleData[{"Audio","NoisyTalk"},"Audio"];
nonVoicedIntervals=AudioIntervals[a,#RMSAmplitude<.02&&#SpectralFlatness>.0001&,.1,PartitionGranularity->{.06,.01}]

AudioPlot[a,Epilog->{RGBColor[1,0,0,.3],Rectangle[{#[[1]],-1},{#[[2]],1}]&/@nonVoicedIntervals},ImageSize->Medium]

High-Level Analysis

It is possible to use neural networks–based functions to gain a deeper insight into the contents of a signal.

Some of the neural networks–based functionality.

Perform high-level audio analysis:

a=SpeechSynthesize["hello i am a computer"]

SpeechRecognize[a]

PitchRecognize[a,Method->"CREPE"]//ListLinePlot

AudioIdentify[a]

All the machine learning functions are aware of (RefLink[Audio, paclet:ref/Audio]) objects and perform their computations starting with a semantically significant feature extraction.

Some of the high-level machine learning functions.

This preprocessing transforms each audio object in a fixed-size vector, so that they can be easily compared.

Classify a small dataset of instrument sounds:

data=Join@@Table[Thread[WebAudioSearch[instr,"Samples",Sequence[Slot["Duration"]<5&,MaxItems->40]]->instr],{instr,{"acoustic guitar","drums","trumpet"}}];
{train,test}=TakeDrop[RandomSample[data],90];

cl=Classify[train]

ClassifierMeasurements[cl,test,"Report"]

Plot the features of a audio collection using (RefLink[FeatureSpacePlot, paclet:ref/FeatureSpacePlot]).

Plot a list of signals in a semantically meaningful space with (RefLink[FeatureSpacePlot, paclet:ref/FeatureSpacePlot]):

FeatureSpacePlot[ExampleData[#]->#[[2]]&/@ExampleData["Audio"],LabelingFunction->Callout]

Neural Networks

The (RefLink[Audio, paclet:ref/Audio]) object is tightly integrated in the powerful neural networks framework. (RefLink[NetEncoder, paclet:ref/NetEncoder]) provides an easy entry point into neural nets for various high-level constructs such as the (RefLink[Audio, paclet:ref/Audio]) object.

Some examples of audio (RefLink[NetEncoder, paclet:ref/NetEncoder]).

Different encoders can be used to compute various kinds of features. Some maintain all of the information of the original signal (like (StringTypeLink["Audio", paclet:ref/netencoder/Audio]) and (StringTypeLink["AudioSTFT", paclet:ref/netencoder/AudioSTFT])), while others provide a compromise between discarding some information but dramatically reducing the dimensionality (like (StringTypeLink["AudioMFCC", paclet:ref/netencoder/AudioMFCC])).

Visualize features computed using different encoders:

a=ExampleData[{"Audio","Bird"},"Audio"]

Labeled[ListLinePlot[First@Transpose@Normal[NetEncoder["Audio"][a]]],"Audio"]

Labeled[MatrixPlot[Log@Transpose@Normal[NetEncoder["AudioSpectrogram"][a]],DataReversed->{True,False}],"AudioSpectrogram"]

Labeled[MatrixPlot[Transpose@Normal[NetEncoder["AudioMFCC"][a]],DataReversed->{True,False}],"AudioMFCC"]

Using the encoders, it is easy to train networks from scratch to solve audio-related tasks and produce measurements of the resulting performance.

Some of the neural networks functionality.

Use (RefLink[NetChain, paclet:ref/NetChain]) and (RefLink[NetGraph, paclet:ref/NetGraph]) to create networks of arbitrary topology, and leverage sequence-focused layers such as (RefLink[GatedRecurrentLayer, paclet:ref/GatedRecurrentLayer]) and (RefLink[LongShortTermMemoryLayer, paclet:ref/LongShortTermMemoryLayer]) to analyze variable length signals.

Train a network to classify spoken digits from 0 to 9:

net=NetChain[{GatedRecurrentLayer[64],GatedRecurrentLayer[64],GatedRecurrentLayer[10],AggregationLayer[Mean,1],SoftmaxLayer[]},"Input"->NetEncoder["AudioMFCC"],"Output"->NetDecoder[{"Class",Range[0,9]}]]

trainedNet=NetTrain[net,ResourceData["Spoken Digit Commands"],ValidationSet->Scaled[.05]]

NetMeasurements[trainedNet,ResourceData["Spoken Digit Commands","TestDataset"],{"Accuracy","ConfusionMatrixPlot"}]