Introduction to Neural Nets

LeNet and MNIST

This tutorial gives a brief overview of the Wolfram Language neural net framework by showing how to train a net that takes an input image of a handwritten single-digit number and then predicts the number. The dataset we are training on is the classic MNIST dataset, and we will train a variant of LeNet, one of the first convolutional nets, which (WebLink[is already available, https://resources.wolframcloud.com/NeuralNetRepository/resources/LeNet-Trained-on-MNIST-Data]) in the (WebLink[Wolfram Neural Net Repository, http://resources.wolframcloud.com/NeuralNetRepository]).

Obtain the MNIST dataset, which contains 60,000 training and 10,000 test images:

trainingData=ResourceData["MNIST","TrainingData"];
testData=ResourceData["MNIST","TestData"];

Display a few random examples from the training set:

RandomSample[trainingData,5]

Obtain a pre-trained version of LeNet from the Wolfram Neural Net Repository:

lenetModel=NetModel["LeNet Trained on MNIST Data"]

Classify a list of images using the pre-trained net:

lenetModel[{,,,}]

It is extremely easy to train a network like LeNet from scratch. (RefLink[NetTrain, paclet:ref/NetTrain]) takes care of many details of the training process automatically, such as selecting an appropriate loss function, attaching encoders and decoders and choosing a batch size. Here is what it looks like.

Train LeNet from scratch:

NetTrain[NetModel["LeNet"],trainingData,ValidationSet->testData]

However, this is not the end of the tutorial.

To give an overview of the fundamentals of deep learning in the Wolfram Language, we will now do this the hard way, by building LeNet out of its component layers, picking a loss function, defining a training network, attaching encoders and decoders, and finally training and evaluating the network. Understanding the general principles behind this particular task will put you well on the way to wielding the Wolfram Language to tackle sophisticated learning tasks easily and efficiently.

Layers

The simplest building blocks of neural nets are layers, which you can think of as simple functions that transform arrays of numbers.

Have a look at the pre-trained LeNet model again (we clicked the FEBitmapsSquarePlusIcon button on the display form to show the constituent layers):

lenetModel

The net is composed of a variety of layers, such as a (RefLink[ConvolutionLayer, paclet:ref/ConvolutionLayer]), (RefLink[PoolingLayer, paclet:ref/PoolingLayer]), etc. Each of these layers accomplishes different tasks, in this case, tasks related to computer vision.

Take a look at the last layer of the net.

Extract the last layer of the net using (RefLink[NetExtract, paclet:ref/NetExtract]):

layer=NetExtract[lenetModel,-1]

We can see a lot of information about this layer. For example, it expects as input a vector of length 10 and returns the same.

Like any layer, we can apply this layer to an input to get an output:

layer[{1,2,5,2,4,7,4,3,5,3}]

A layer can also accept a (RefLink[NumericArray, paclet:ref/NumericArray]) input, yielding a (RefLink[NumericArray, paclet:ref/NumericArray]) in this case:

layer[NumericArray[]]

Normal[%]

The purpose of a (RefLink[SoftmaxLayer, paclet:ref/SoftmaxLayer]) is to produce probabilities that sum to 1.

Sum up the previous output:

Total[%]

Let us construct a new layer.

Create a (RefLink[LinearLayer, paclet:ref/LinearLayer]) that takes as input a vector of length 2 and produces as output a vector of length 3:

linear=LinearLayer[3,"Input"->2]

Notice that in the summary box above, there is an "uninitialized" caption indicating that the net contains learnable parameters that have not yet been provided.

Apply the uninitialized layer to an input vector. This will fail:

linear[{2,3}]

Only certain layers, for example, (RefLink[ConvolutionLayer, paclet:ref/ConvolutionLayer]) and (RefLink[LinearLayer, paclet:ref/LinearLayer]), have learnable parameters. Such layers always have the ({{}}) icon in the display form. Layers with the icon ({{}}), by contrast, do not contain any learnable parameters.

We can supply the random values for the learnable parameters using (RefLink[NetInitialize, paclet:ref/NetInitialize]).

Initialize the layer:

initializedLinear=NetInitialize[linear]

Apply the initialized layer to an input vector:

initializedLinear[{2,3}]

Obtain the weights and biases from the initialized layer:

NetExtract[initializedLinear,"Weights"]

NetExtract[initializedLinear,"Biases"]

Notice that weights and biases parameters of the layers are picked in (RefLink[NumericArray, paclet:ref/NumericArray]). They can be converted to a list using (RefLink[Normal, paclet:ref/Normal]):

Normal@%

So far, we have seen layers that have exactly one input. Some layers have more than one input. For example, (RefLink[MeanSquaredLossLayer, paclet:ref/MeanSquaredLossLayer]) compares two arrays, called the input and the target, and produces a single number that represents (RefLink[Mean, paclet:ref/Mean])[(input-target)^2].

Create a (RefLink[MeanSquaredLossLayer, paclet:ref/MeanSquaredLossLayer]):

msloss=MeanSquaredLossLayer[]

The inputs of the layer are named and must be supplied in an association when the net is applied.

Apply the layer to two inputs:

msloss[<|"Input"->{1,2,3},"Target"->{4,0,4}|>]

While some layers introduce functionality that is unique to the neural net framework, others mirror the functionality of existing Wolfram Language symbols. For example, (RefLink[FlattenLayer, paclet:ref/FlattenLayer]) behaves similarly to (RefLink[Flatten, paclet:ref/Flatten]), (RefLink[DotLayer, paclet:ref/DotLayer]) behaves similarly to (RefLink[Dot, paclet:ref/Dot]), etc.

The full list of available layers is:

?*Layer

More Properties of Layers (Advanced)

This section summarizes a few key properties of neural net layers in the Wolfram Language.

Layers are functions whose inputs and outputs are always arrays or scalars. In order to operate on audio, images, etc., layers must have a (RefLink[NetEncoder, paclet:ref/NetEncoder]) attached to the input.

Layers perform shape inference: if the dimensions of the input are specified, then the layer can infer what the dimensions of the output will be, and vice versa.

Layers are differentiable. Differentiability is a key property that allows for the efficient training of nets, which we will see later.

Layers can be run on either a CPU or an NVIDIA GPU, by specifying the (RefLink[TargetDevice, paclet:ref/TargetDevice]) option.

By default, layers evaluate using lower numerical precision (single-precision floating point) than is typical in Wolfram Language functions. The (RefLink[WorkingPrecision, paclet:ref/WorkingPrecision]) option can be used to specify different behavior.

Net Encoders

Fundamentally, because they must be differentiable, neural net layers operate on arrays. However, we often want to train and use nets on other data, such as images, audio, text, etc. To do this, we can use a (RefLink[NetEncoder, paclet:ref/NetEncoder]) to translate this data to an array of values.

To translate the images of digits in the MNIST dataset, we can use an (StringTypeLink["Image", paclet:ref/netencoder/Image]) encoder. Let us first look at some simple examples of encoders in action.

Create an image (RefLink[NetEncoder, paclet:ref/NetEncoder]) that produces a 1×12×12 array:

imageenc=NetEncoder[{"Image",{12,12},"ColorSpace"->"Grayscale"}]

Apply the image (RefLink[NetEncoder, paclet:ref/NetEncoder]) to an image:

imageenc[]//Normal

The image encoder conforms the image to have the specified colorspace, dimensions, etc. before it is converted to an array.

Apply the image (RefLink[NetEncoder, paclet:ref/NetEncoder]) on a large color image:

img=;
ImageDimensions[img]

imageenc[img]//Dimensions

While encoders can be used independently from nets, as we have done, it is more common to attach the encoder to a layer. This can be done when creating the layer or afterward. Here is an example of creating a layer with an attached encoder.

Attach an image (RefLink[NetEncoder, paclet:ref/NetEncoder]) to a (RefLink[PoolingLayer, paclet:ref/PoolingLayer]) via the "Input" option:

pool=PoolingLayer[10,"Input"->NetEncoder[{"Image",64}]]

Apply the (RefLink[PoolingLayer, paclet:ref/PoolingLayer]) directly to an image, which will use the image (RefLink[NetEncoder, paclet:ref/NetEncoder]) to translate the image to an array for (RefLink[PoolingLayer, paclet:ref/PoolingLayer]) to operate on:

ArrayPlot/@pool[]

Convert the output back to an image:

Image[pool[],Interleaving->False]

The actual images in MNIST are grayscale images of size 28×28. Let us create our final image encoder now. Later, we will attach this (RefLink[NetEncoder, paclet:ref/NetEncoder]) when we construct LeNet from scratch.

Create an (StringTypeLink["Image", paclet:ref/netencoder/Image]) encoder for MNIST:

mnistEncoder=NetEncoder[{"Image",{28,28},"ColorSpace"->"Grayscale"}]

The dimensions of this encoder match the dimensions of the images in the MNIST dataset:

trainingData[[1,1]]

ImageDimensions[%]

Net Decoders

The output of a neural net is often a prediction. For a regression problem, this prediction is typically a point estimate, which means it is a single number representing the value the net thinks is most likely for the task. Such outputs do not typically need to be decoded.

For classification problems, however, the output of the net is typically a vector whose components represent the probability of each class. For example, a net that classifies images of foods as "hot dog", "pizza", or "salad" produces a vector with three components that sum to one, representing the probabilities of those three classes.

For these kinds of probability vector outputs, we typically care about the most likely class rather than the raw probabilities. To determine this, we have to know how classes are associated with particular vector components.

There are also other properties we could also compute from a probability vector, such as the top n probabilities (if we have many classes), the probability of a specific class, or a measure of the uncertainty of the prediction.

To make these kinds of queries more convenient, a (StringTypeLink["Class", paclet:ref/netdecoder/Class]) (RefLink[NetDecoder, paclet:ref/NetDecoder]) can be used to store the mapping between vector components and classes and thus automatically interpret the output of the net. Other types of (RefLink[NetDecoder, paclet:ref/NetDecoder]) are also possible, for converting the output to an (RefLink[Image, paclet:ref/Image]), a Boolean value, etc., though we do not discuss them in detail in this tutorial.

For the MNIST task, the 10 classes we will use are the digits 0 through 9. Let us create an appropriate decoder.

Create a (StringTypeLink["Class", paclet:ref/netdecoder/Class]) (RefLink[NetDecoder, paclet:ref/NetDecoder]) to interpret a vector of probabilities:

mnistDecoder=NetDecoder[{"Class",Range[0,9]}]

By default, the decoder will decode a probability vector as a most likely class (if this looks confusing, recall that the first class is actually 0).

Apply the decoder to a probability vector:

mnistDecoder[{0,0,0,1,0,0,0,0,0,0}]

We can also compute other properties, by supplying a named property as the second argument when applying the net.

Obtain a list of the most likely classes and their probabilities:

mnistDecoder[{0,0,.3,.3,.3,.1,0,0,0,0},"TopProbabilities"]

Obtain the probability of a specific class:

mnistDecoder[{0,0,.3,.3,.3,.1,0,0,0,0},{"Probability",2}]

Obtain the full list of probabilities as an association:

mnistDecoder[{0,0,.3,.3,.3,.1,0,0,0,0},"Probabilities"]

Obtain a measure of the uncertainty of the prediction:

mnistDecoder[{0,0,.3,.3,.3,.1,0,0,0,0},"Entropy"]

As with (RefLink[NetEncoder, paclet:ref/NetEncoder]), we can attach a (RefLink[NetDecoder, paclet:ref/NetDecoder]) to the output of a layer. Here is a more streamlined example of the (RefLink[PoolingLayer, paclet:ref/PoolingLayer]) we showed before, in which an (RefLink["Image", paclet:ref/NetEncoder/Image]) (RefLink[NetEncoder, paclet:ref/NetEncoder]) is used to interpret the input to the layer, and an (RefLink["Image", paclet:ref/NetDecoder/Image]) (RefLink[NetDecoder, paclet:ref/NetDecoder]) is used to convert the final output of the layer back to an image.

Attach both a (RefLink[NetEncoder, paclet:ref/NetEncoder]) and a (RefLink[NetDecoder, paclet:ref/NetDecoder]) to a layer:

pool=PoolingLayer[10,"Input"->NetEncoder[{"Image",64}],"Output"->NetDecoder["Image"]]

Applying the layer to an image will produce an image:

pool[]

Containers

Single neural net layers are generally not useful by themselves. We usually need to combine multiple layers together to do something interesting.

The simplest way to combine layers is chain them one after another, where the output of the first layer is used as the input for the next layer, and so on. We can use the (RefLink[NetChain, paclet:ref/NetChain]) container to connect layers in this way, but when more complex forms of connectivity are required, the (RefLink[NetGraph, paclet:ref/NetGraph]) container should be used instead.

For now, let us create a simple chain.

Create a simple (RefLink[NetChain, paclet:ref/NetChain]) computing (RefLink[Cos, paclet:ref/Cos])[(RefLink[Sin, paclet:ref/Sin])[x]]:

chain=NetChain[{ElementwiseLayer[Sin],ElementwiseLayer[Cos]}]

Apply the (RefLink[NetChain, paclet:ref/NetChain]) to an input and compare the result to applying (RefLink[Sin, paclet:ref/Sin]) followed by (RefLink[Cos, paclet:ref/Cos]) to the same input:

chain[{3.2,-2.3}]

Cos[Sin[{3.2,-2.3}]]

Create a simple (RefLink[NetChain, paclet:ref/NetChain]) that consists of an (RefLink[ElementwiseLayer, paclet:ref/ElementwiseLayer]) that applies the (RefLink[Ramp, paclet:ref/Ramp]) function, followed by a (RefLink[SoftmaxLayer, paclet:ref/SoftmaxLayer]):

layer1=ElementwiseLayer[Ramp];
layer2=SoftmaxLayer[];
chain=NetChain[{layer1,layer2}]

Apply the (RefLink[NetChain, paclet:ref/NetChain]) to an input:

chain[{-1,3,2,4}]

The result is the same as applying the individual layers in succession:

layer1[{-1,3,2,4}]

layer2[%]

An important property of containers is that they act as layers and can even be used as layers within other containers. Let us see an example of that.

Nest the previous (RefLink[NetChain, paclet:ref/NetChain]) inside another (RefLink[NetChain, paclet:ref/NetChain]):

chain2=NetChain[{ElementwiseLayer[Sin],ElementwiseLayer[Cos],chain}]

Apply the chain to an input:

chain2[{-1,3,2,4}]

Flatten the nested chains together:

NetFlatten[chain2]

Previously, we saw the LeNet model, which was a more complex chain. Here is the code that constructs an uninitialized copy of LeNet.

Construct LeNet from scratch, supplying the previously constructed (RefLink[NetEncoder, paclet:ref/NetEncoder]) and (RefLink[NetDecoder, paclet:ref/NetDecoder]):

uninitializedLenet=NetChain[{ConvolutionLayer[20,5],ElementwiseLayer[Ramp],PoolingLayer[2,2],ConvolutionLayer[50,5],ElementwiseLayer[Ramp],PoolingLayer[2,2],FlattenLayer[],LinearLayer[500],ElementwiseLayer[Ramp],LinearLayer[10],SoftmaxLayer[]},"Input"->mnistEncoder,"Output"->mnistDecoder]

Note that the layers containing learnable parameters appear in red, indicating that they require initial values before the net can be applied to an input.

As a quick exercise, we will now randomly initialize LeNet and apply it to a sample input from MNIST. The output we get is, of course, also random, but serves to illustrate that things are working properly.

Randomly initialize the learnable parameters of LeNet:

lenet=NetInitialize[uninitializedLenet]

Apply the initialized LeNet to an input image, producing a random classification:

lenet[]

Obtain a sorted list of the top probabilities:

lenet[,"TopProbabilities"]

Our ultimate goal, of course, is to teach this randomly initialized network to correctly classify handwritten digits.

Graphs

In order to train LeNet, we need to construct a training network that feeds individual training examples to LeNet. Each training example in the MNIST dataset consists of the combination of an input image and a corresponding target label.

Show a set of examples from MNIST:

RandomSample[trainingData,4]

(RefLink[NetChain, paclet:ref/NetChain]) does not allow a net to take more than one input, so we need to use (RefLink[NetGraph, paclet:ref/NetGraph]) to build the training network. The task of the training network is to evaluate the prediction produced by LeNet, producing a small number if the prediction is good, and a large number if it is not. This is called a loss. It is best to think of it as a sort of proxy for prediction error.

Once we have this training network, we can use the (RefLink[NetTrain, paclet:ref/NetTrain]) function to gradually modify the learnable parameters in the net so that the loss decreases over time.

For different learning tasks, different ways of computing the loss must be used. For a classification task such as classifying MNIST digits, a common choice of loss is the cross entropy loss. The layer (RefLink[CrossEntropyLossLayer, paclet:ref/CrossEntropyLossLayer]) can compute this loss when given both a prediction and a true label, or target.

The prediction we will use is in the form of a vector of probabilities, where each element of the vector represents the probability of the corresponding digit 0, 1, 2, etc. The target label is the index of the correct class (1 for digit 0, 2 for digit 1, etc.).

Here is a simple example of a (RefLink[CrossEntropyLossLayer, paclet:ref/CrossEntropyLossLayer]) being used to score a prediction.

Create a (RefLink[CrossEntropyLossLayer, paclet:ref/CrossEntropyLossLayer]) that compares an input prediction vector of length 5 with a target label:

loss=CrossEntropyLossLayer["Index","Input"->{5}]

Apply the loss layer to a prediction vector in which the first component has a large probability:

probvector={0.8,0.05,0.05,0.05,0.05}

If the target is 1, the loss will be low, as the prediction has assigned high probability to the class 1:

loss[<|"Input"->probvector,"Target"->1|>]

If the target is 5, the loss will be high, as the prediction has assigned low probability to class 5:

loss[<|"Input"->probvector,"Target"->5|>]

The training net we will construct is simple: it applies LeNet to the image of the digit to produce a prediction, and then it compares this prediction with the target class.

Construct a (RefLink[NetGraph, paclet:ref/NetGraph]) by supplying a list of layers and connections. Inputs to the graph are connected to layers using the syntax (RefLink[NetPort, paclet:ref/NetPort])["input"]->destination, and the inputs to the loss layer are connected via source->(RefLink[NetPort, paclet:ref/NetPort])["loss","input"]:

trainingNet=NetGraph[<|"lenet"->lenet,"loss"->CrossEntropyLossLayer["Index"]|>,
{NetPort["Input"]->"lenet"->NetPort["loss","Input"],NetPort["Target"]->NetPort["loss","Target"]}]

To test things out, let us use our training net, which contains a randomly initialized prediction LeNet, on a set of inputs and corresponding targets.

Feed a list of images to the "Input" port and a list of indices to the "Target" port:

trainingNet[\[LeftAssociation]"Input"->{,,,,},"Target"->{2,4,7,9,8}\[RightAssociation]]

These losses summarize how well LeNet did at predicting the targets when given the images. Because LeNet was randomly initialized, we expect it to be no better than chance (on average). During training, the learnable parameters in LeNet are gradually adjusted to bring the average loss down.

How is this accomplished? The key idea is via gradients. These are adjustments to the learnable parameters that can be calculated through a process called backpropagation. These adjustments are derived so as to slightly reduce the average loss of the training net on a specific batch of examples.

By repeatedly selecting a batch of examples at random, calculating the adjustment, and applying it to the learnable parameters, the net gradually improves at the desired task.

This process is handled by (RefLink[NetTrain, paclet:ref/NetTrain]), which offers many ways to adjust and fine-tune the training process. But we can get some insight into the mechanism involved by calculating one of these gradients directly, using (RefLink[NetPortGradient, paclet:ref/NetPortGradient]).

Request the gradient produced by a specific input at the biases of the first convolution layer of LeNet:

gradient=trainingNet[\[LeftAssociation]"Input"->,"Target"->2\[RightAssociation],NetPortGradient[{"lenet",1,"Biases"}]]

Now that we have the gradient, we can actually modify the corresponding learnable parameter using this gradient, which reduces the loss on this example.

Modify the corresponding value of the training net by obtaining the bias value, adjusting it slightly using the gradient, and replacing it in the original net:

oldBias=Normal@NetExtract[trainingNet,{"lenet",1,"Biases"}];
newBias=oldBias-0.1*gradient;
trainingNet2=NetReplacePart[trainingNet,{"lenet",1,"Biases"}->newBias]

Compare the loss on the modified net with the original net. The loss has decreased:

trainingNet[\[LeftAssociation]"Input"->,"Target"->2\[RightAssociation]]

trainingNet2[\[LeftAssociation]"Input"->,"Target"->2\[RightAssociation]]

Training

We are now ready to train our network with (RefLink[NetTrain, paclet:ref/NetTrain]).

Normally, (RefLink[NetTrain, paclet:ref/NetTrain]) performs the construction of the training network automatically. For simple networks with one input and one output, it handles training data of the form {in->out,…} by feeding the (Subscript[in, i]) to the net and then comparing the output of the net with the (Subscript[out, i]) via a suitable loss function.

But because we have explicitly constructed a training network, we must provide the training data in the form <|"port"->list,…|>, where we explicitly feed the input ports of the training net with lists of data. So we must first convert the data we have, which is in the form of rules in->out.

As one additional complication, we must also account for the fact that the training data contains labels that are integers 0 to 9, whereas the "Target" input of our training net is expecting an index in the range 1 to 10. We could use a (StringTypeLink["Class", paclet:ref/netencoder/Class]) (RefLink[NetEncoder, paclet:ref/NetEncoder]) to convert them (which (RefLink[NetTrain, paclet:ref/NetTrain]) would normally automate), but instead we will exploit the fact that we can just add 1 to accomplish the same thing.

Convert the training and test data into association form, using (RefLink[Keys, paclet:ref/Keys]) and (RefLink[Values, paclet:ref/Values]) to obtain the images and the labels from the lists of rules in the training and test data:

trainAssoc=<|"Input"->Keys[trainingData],"Target"->Values[trainingData]+1|>;
testAssoc=<|"Input"->Keys[testData],"Target"->Values[testData]+1|>;

Show a small sample of the training association:

Take[trainAssoc,All,5]

Let us now perform the training using (RefLink[NetTrain, paclet:ref/NetTrain]). Notice several things about this net example:

We have specified (RefLink[MaxTrainingRounds, paclet:ref/MaxTrainingRounds])->5, which will scan the entire training set five times before finishing.

We have given (RefLink[All, paclet:ref/All]) as the third argument, to obtain a (RefLink[NetTrainResultsObject, paclet:ref/NetTrainResultsObject]) that summarizes various information about the training session.

We have given a (RefLink[ValidationSet, paclet:ref/ValidationSet]), which allows us to measure how well the trained classifier generalizes to new examples that it has not trained on. This helps avoid a common pitfall known as "overfitting".

We have specified (RefLink[TargetDevice, paclet:ref/TargetDevice])->"CPU" (which is already the default). If you have an NVIDIA graphics card, however, you can change this to "GPU" to achieve a big speedup in training.

Train LeNet:

results=NetTrain[trainingNet,trainAssoc,All,ValidationSet->testAssoc,MaxTrainingRounds->5,TargetDevice->"CPU"]

The final (RefLink[NetTrainResultsObject, paclet:ref/NetTrainResultsObject]) gives us a wealth of information. Some of it is shown in its display form, but more information can be retrieved from it programmatically. But we can immediately see that our network was able to achieve an accuracy of about 99.4% after around 3 minutes of CPU training.

Let us have another look at the evolution of the loss.

Obtain the loss evolution plot from the (RefLink[NetTrainResultsObject, paclet:ref/NetTrainResultsObject]):

results["LossPlot"]

Perhaps most importantly, we can extract the final network.

Obtain the trained net from the (RefLink[NetTrainResultsObject, paclet:ref/NetTrainResultsObject]) and from it extract the prediction network:

trainedLenet=NetExtract[results["TrainedNet"],"lenet"]

Reattach the (RefLink[NetEncoder, paclet:ref/NetEncoder]) and (RefLink[NetDecoder, paclet:ref/NetDecoder]) that were removed when the classification net was embedded inside the training network:

trainedLenet=NetReplacePart[trainedLenet,{"Input"->mnistEncoder,"Output"->mnistDecoder}]

The trained network can now perform classifications.

Make a classification on an image:

trainedLenet[]

Obtain the top probabilities for a difficult image:

trainedLenet[,"TopProbabilities"]

That is it! We have built and trained LeNet from scratch, and in the process covered the following topics:

An introduction to neural net layers and how to combine them with (RefLink[NetChain, paclet:ref/NetChain]) and (RefLink[NetGraph, paclet:ref/NetGraph]).

How to use (RefLink[NetEncoder, paclet:ref/NetEncoder]) and (RefLink[NetDecoder, paclet:ref/NetDecoder]) to translate between arrays and other forms of data.

How to build a training net that computes the loss of our prediction net using (RefLink[CrossEntropyLossLayer, paclet:ref/CrossEntropyLossLayer]).

How to call (RefLink[NetTrain, paclet:ref/NetTrain]) to train a net using training data.

In the next section, we cover how to evaluate a trained net.

Evaluation

Now that we have trained our net, we can derive a lot more information about it. For example, we can obtain the overall accuracy on another dataset. We can also get useful summaries like the confusion matrix, which summarizes how the net misclassifies examples. And we can compare the performance of the net, which uses deep learning, to that of other common machine learning techniques.

Let us jump in by building a (RefLink[ClassifierMeasurementsObject, paclet:ref/ClassifierMeasurementsObject]) that measures various properties of the net's classification behavior on a dataset. It is important to use a dataset we did not train on, so we use the test set that comes built into the MNIST dataset.

Use (RefLink[ClassifierMeasurements, paclet:ref/ClassifierMeasurements]) to obtain a (RefLink[ClassifierMeasurementsObject, paclet:ref/ClassifierMeasurementsObject]) on the test set:

measurements=ClassifierMeasurements[trainedLenet,testData]

Now that we have the (RefLink[ClassifierMeasurementsObject, paclet:ref/ClassifierMeasurementsObject]), we can query for all sorts of properties efficiently.

Obtain the overall accuracy:

measurements["Accuracy"]

Obtain a list of examples for which the loss was the highest:

measurements["WorstClassifiedExamples"]

Obtain a list of examples for which the net was least certain of the class:

measurements["LeastCertainExamples"]

Obtain a list of which misclassifications are most common:

measurements["TopConfusions"]

Obtain the full set of available properties:

measurements["Properties"]

We can also use (RefLink[NetMeasurements, paclet:ref/NetMeasurements]) to measure various properties of the net. (RefLink[NetMeasurements, paclet:ref/NetMeasurements]) and (RefLink[ClassifierMeasurements, paclet:ref/ClassifierMeasurements]) share many of the same properties, but each have a few of their own. (RefLink[NetMeasurements, paclet:ref/NetMeasurements]) has the advantage that it is efficiently implemented and even can be run on a GPU, using the (RefLink[TargetDevice, paclet:ref/TargetDevice]) option, which makes it much more applicable for large datasets.

Obtain the overall accuracy:

NetMeasurements[trainedLenet,testData,"Accuracy"]

Obtain a confusion matrix:

NetMeasurements[trainedLenet,testData,"ConfusionMatrixPlot"]

Obtain a list of scores of how well various classes are classified:

NetMeasurements[trainedLenet,testData,"F1Score"]

(RefLink[NetMeasurements, paclet:ref/NetMeasurements]) has a few other advantages for neural networks. It is possible to measure the output and weights of any layer in a network.

Plot the mean activations of each filter in the first layer of our trained LeNet model:

activations=NetMeasurements[trainedLenet,testData,NetPort[{1,"Output"}]];
ArrayPlot[ArrayFlatten @ Partition[activations,5],ColorFunction->"Rainbow",FrameTicks->None]

Additionally, (RefLink[NetMeasurements, paclet:ref/NetMeasurements]) uses caching to speed up the repeated measurement of the same properties or even the measurement of similar properties.

Time the measurement of Cohen's Kappa, which will be cached due to measuring "ConfusionMatrixPlot" above. Note that it takes a fraction of a second to make 10000 measurements:

NetMeasurements[trainedLenet,testData,"CohenKappa"]//AbsoluteTiming

Finally, (RefLink[NetMeasurements, paclet:ref/NetMeasurements]) can be used in more cases than (RefLink[ClassifierMeasurements, paclet:ref/ClassifierMeasurements]): it can be used when the net has none of the required loss layers and no (RefLink[NetEncoder, paclet:ref/NetEncoder]) or (RefLink[NetDecoder, paclet:ref/NetDecoder]) attached, as well as when the net has multiple outputs.

The easiest way to compare the net to other methods is to use (RefLink[Classify, paclet:ref/Classify]), which automatically applies a range of common methods and chooses the best.

Use (RefLink[Classify, paclet:ref/Classify]) to automatically pick an effective machine learning model:

classifier=Classify[trainingData,ValidationSet->testData]

Apply the classifier to an input:

classifier[,"Probabilities"]

The neural net outperforms the model selected by (RefLink[Classify, paclet:ref/Classify]) on the test set.

Compare the classifier with the trained net:

NetMeasurements[trainedLenet,testData,"Accuracy"]

ClassifierMeasurements[classifier,testData,"Accuracy"]