Computer Vision

Image Classification

MNIST Digit Classification

Train a digit recognizer on the MNIST database of handwritten digits using a convolutional neural network.

First, obtain the training and validation data:

resource = ResourceObject["MNIST"]

trainingData=ResourceData[resource,"TrainingData"];
testData=ResourceData[resource,"TestData"];

RandomSample[trainingData,5]

Define a convolutional neural network that takes in 28×28 grayscale images as input:

lenet=NetChain[{
ConvolutionLayer[20,5],Ramp,PoolingLayer[2,2],
ConvolutionLayer[50,5],Ramp,PoolingLayer[2,2],
FlattenLayer[],500,Ramp,10,SoftmaxLayer[]},
"Output"->NetDecoder[{"Class",Range[0,9]}],
"Input"->NetEncoder[{"Image",{28,28},"Grayscale"}]
]

Train the network for three training rounds. (RefLink[NetTrain, paclet:ref/NetTrain]) will automatically attach a (RefLink[CrossEntropyLossLayer, paclet:ref/CrossEntropyLossLayer]) using the same classes that were provided to the decoder:

results=NetTrain[lenet,trainingData,All,ValidationSet->testData,MaxTrainingRounds-> 3]
trained=results["TrainedNet"]

Evaluate the trained network directly on images randomly sampled from the validation set:

imgs=Keys @ RandomSample[testData,5];
Thread[imgs->trained[imgs]]

Obtain probability estimates of all classes for a specific input:

trained[,"Probabilities"]

Use (RefLink[NetMeasurements, paclet:ref/NetMeasurements]) to test the classification performance of the trained net on the test set:

NetMeasurements[trained,testData,"Accuracy"]

NetMeasurements[trained,testData,"ConfusionMatrixPlot"]

CIFAR-10 Object Classification

Using the CIFAR-10 database of labeled images, train a convolutional net to predict the class of each object. First, obtain the training data:

obj=ResourceObject["CIFAR-10"];
trainingData=ResourceData[obj,"TrainingData"];
RandomSample[trainingData,5]

Extract the unique classes:

classes=Union@Values[trainingData]

Create a convolutional net that predicts the class, given an image:

lenet=NetChain[
{ConvolutionLayer[20,5],Ramp,PoolingLayer[2,2],ConvolutionLayer[50,5],Ramp,PoolingLayer[2,2],FlattenLayer[],500,Ramp,10,SoftmaxLayer[]},
"Output"->NetDecoder[{"Class",classes}],
"Input"->NetEncoder[{"Image",{32,32}}]
]

Train the net on the training data:

results=NetTrain[lenet,trainingData,All,MaxTrainingRounds->4]
trained=results["TrainedNet"]

Predict the most likely classes for a set of images:

trained[{,,,,}]

For a specific example, give the probabilities of the most likely label assignments:

trained[,{"TopProbabilities",3}]

From a random sample, select the images for which the net produces highest and lowest entropy predictions. High-entropy inputs can be interpreted as those for which the net is most uncertain about the correct class:

images=RandomSample[Keys[trainingData],5000];

entropies=trained[images,"Entropy"];

Labeled[images[[Ordering[entropies,-10]]],"high entropy"]
Labeled[images[[Ordering[entropies,10]]],"low entropy"]

Learned Embedding

Learn an embedding of the digits in the MNIST dataset.

First, import the data and take only those examples with labels between 0 and 4:

mnist=Select[ResourceData["MNIST"],Last[#]<5&];

Create a training set by sampling pairs of images and associating them with (RefLink[True, paclet:ref/True]) if their labels are different and (RefLink[False, paclet:ref/False]) if their labels are the same:

toRule[{img1_->l1_,img2_->l2_}]:={img1,img2}->l1≠l2;

trainingData=Table[toRule@RandomSample[mnist,2],10000];
Take[trainingData,4]

Define a convolutional net to use as an embedding network:

lenet=NetChain[{
ConvolutionLayer[10,4],Ramp,PoolingLayer[2,2],
ConvolutionLayer[20,4],Ramp,PoolingLayer[2,2],
FlattenLayer[],2},
"Input"->NetEncoder[{"Image",{28,28},"Grayscale"}]]

Construct the embedding net:

net=NetPairEmbeddingOperator[lenet]

Train the network:

results=NetTrain[net,trainingData,All]
trained=results["TrainedNet"]

Apply the network to a list of pairs of digits to compute their distances under the embedding. Digits with the same label have small distances:

trained[{{,},{,},{,}}]

Extract the embedding network:

embedding=NetExtract[trained,"Net"]

Compute the embedding of a digit:

embedding[]

Sample 500 digits and group them by their labels:

groups=KeySort@GroupBy[RandomSample[mnist,2000],Last->First];

Compute their embeddings and plot them. Digits with the same label are clustered under the learned embedding:

points=Map[embedding/*Normal,groups];

ListPlot[Table[points[[i]],{i,1,5}],PlotLegends->Range[0,4],Axes->False,FrameTicks->False,Frame->True,
PlotStyle->PointSize[0.012],AspectRatio->1]

Style Transfer

Create a new image with the content of one image and in the style of another image. This implementation follows the method described in Gatys et al., "A Neural Algorithm of Artistic Style".

An example content and style image:

contentImg=;
styleImg=;

To create the image that is a mix of both of these images, start by obtaining a pre-trained image classification network:

vggNet=NetModel["VGG-16 Trained on ImageNet Competition Data"];

Take a layer that will be used as a feature extractor for the style and content images:

featureNet=Take[vggNet,{1,"relu4_1"}]

There are three loss functions used. The first loss ensures that the content of the synthesized image is similar to that of the content image:

contentLoss=NetGraph[{MeanSquaredLossLayer[]},{1->NetPort["LossContent"]}]

The second loss ensures that the style of the synthesized image is similar to that of the style image. Style similarity is defined as the mean-squared difference between the Gram matrices of the input and target:

gramMatrix=NetGraph[{FlattenLayer[-1],TransposeLayer[1->2],DotLayer[]},{1->3,1->2->3}];

styleLoss=NetGraph[{gramMatrix,gramMatrix,MeanSquaredLossLayer[]},
{NetPort["Input"]->1, NetPort["Target"]->2,{1,2}->3,3->NetPort["LossStyle"]}]

The third loss ensures that the magnitude of intensity changes across adjacent pixels in the synthesized image is small. This helps the synthesized image look more natural:

l2Loss=NetGraph[{ThreadingLayer[(#1-#2)^2&],SummationLayer[]},{{NetPort["Input"],NetPort["Target"]}->1->2}];

tvLoss=NetGraph[<|
"dx1"->PaddingLayer[{{0,0},{1,0},{0,0}},"Padding"->"Fixed" ],
"dx2"->PaddingLayer[{{0,0},{0,1},{0,0}}, "Padding"->"Fixed"],
"dy1"->PaddingLayer[{{0,0},{0,0},{1,0}},"Padding"->"Fixed" ],
"dy2"->PaddingLayer[{{0,0},{0,0},{0,1}}, "Padding"->"Fixed"],
"lossx"->l2Loss,"lossy"->l2Loss,"tot"->TotalLayer[]|>,{
{"dx1","dx2"}->"lossx",{"dy1","dy2"}->"lossy",
{"lossx","lossy"}->"tot"->NetPort["LossTV"]}]

Define a function that creates the final training net for any content and style image. This function also creates a random initial image:

createTransferNet[net_,content_Image,styleFeatSize_]:=Module[
{dims=Prepend[3]@Reverse@ImageDimensions[content]},
NetGraph[<|
"Image"->ConstantArrayLayer["Array"->RandomReal[{-0.1,0.1}, dims]],
"imageFeat"->NetReplacePart[net,"Input"->dims],
"content"->contentLoss,
"style"->styleLoss,
"tv"->tvLoss|>,
{"Image"->"imageFeat",
{"imageFeat",NetPort["ContentFeature"]}->"content",
{"imageFeat",NetPort["StyleFeature"]}->"style",
"Image"->"tv"},
"StyleFeature"->styleFeatSize
]
]

Define a (RefLink[NetDecoder, paclet:ref/NetDecoder]) for visualizing the predicted image:

meanIm=NetExtract[featureNet,"Input"][["MeanImage"]]

decoder=NetDecoder[{"Image","MeanImage"->meanIm}]

The training data consists of features extracted from the content and style images. Define a feature extraction function:

extractFeatures[img_]:=NetReplacePart[featureNet,"Input"->NetEncoder[{"Image",ImageDimensions[img],"MeanImage"->meanIm}]][img];

Create a training set consisting of a single example of a content and style feature:

trainingdata=<|
"ContentFeature"->{extractFeatures[contentImg]},"StyleFeature"->{extractFeatures[styleImg]}
|>;

Create the training net whose input dimensions correspond to the content and style image dimensions:

net=createTransferNet[featureNet,contentImg,Dimensions@First@trainingdata["StyleFeature"]];

When training, the three losses are weighted differently to set the relative importance of the content and style. These values might need to be changed with different content and style images. Create a loss specification that defines the final loss as a combination of the three losses:

perPixel=1/(3*Apply[Times,ImageDimensions[contentImg]]);
lossSpec={"LossContent"->Scaled[6.*10^-5],"LossStyle"->Scaled[0.5*10^-14],"LossTV"->Scaled[20.*perPixel]};

Optimize the image using (RefLink[NetTrain, paclet:ref/NetTrain]). (RefLink[LearningRateMultipliers, paclet:ref/LearningRateMultipliers]) are used to freeze all parameters in the net except for the (RefLink[ConstantArrayLayer, paclet:ref/ConstantArrayLayer]). The training is best done on a GPU, as it will take up to an hour to get good results with CPU training. The training can be stopped at any time via Evaluation ▶ Abort Evaluation:

results=NetTrain[net,
trainingdata,All,LossFunction->lossSpec,
LearningRateMultipliers->{"Image"->1,_->None},
TrainingProgressReporting->Function[decoder[#Weights[{"Image","Array"}]]],
MaxTrainingRounds->300,BatchSize->1,
Method->{"ADAM","LearningRate"->0.05},
TargetDevice->"GPU"
]
trainedNet=results["TrainedNet"]

Extract the final image from the (RefLink[ConstantArrayLayer, paclet:ref/ConstantArrayLayer]) of the trained net:

decoder[NetExtract[trainedNet,{"Image","Array"}]]

Semantic Segmentation (Pixel Classification)

Semantic Segmentation on a Toy Text Dataset

Train a net that classifies every pixel in an image of a word as being part of the background or part of one of the letters a through z.

First, generate training and test data, which consists of images of words and the corresponding "mask" integer matrices that label each pixel:

imgMap=\[LeftAssociation]" "->,"a"->,"b"->,"c"->,"d"->,"e"->,"f"->,"g"->,"h"->,"i"->,"j"->,"k"->,"l"->,"m"->,"n"->,"o"->,"p"->,"q"->,"r"->,"s"->,"t"->,"u"->,"v"->,"w"->,"x"->,"y"->,"z"->\[RightAssociation];

class=0;
maskMap=Map[1+class++*(1-ImageData[Binarize[#,0.7]])&,imgMap];

maxLen=7;
wordList=Select[WordList[],StringLength[#]≤maxLen&&LetterQ[#]&];
wordList=RandomSample@StringPadRight[ToLowerCase[wordList],maxLen];

rules=Map[
ImageAssemble[Lookup[imgMap,#]]->MapThread[Join,Lookup[maskMap,#]]&,
Characters[wordList]];
Length[rules]

{testData,trainData}=TakeDrop[rules,Ceiling[Length[rules]/10]];

Obtain the number of classes:

classNum=Max[maskMap]

Assign a color to each class index:

colorRules = Thread[Range[classNum] -> Prepend[RandomColor[classNum - 1], White]]

Obtain a random sample of input images in the training set:

RandomSample[Keys@trainData,5]

Visualize the mask array of a single example, where each color represents a particular class:

ArrayPlot[trainData[[1005,2]],ColorRules->colorRules]

Define a convolutional net that takes an image and returns a probability vector for every pixel in the image:

basicBlock[channels_,kernelSize_,opts___]:=NetChain[{ConvolutionLayer[channels,kernelSize,opts],BatchNormalizationLayer[],Ramp}]

net=NetChain[{
(* bottleneck layers *)
basicBlock[32,1],
basicBlock[32,1],
basicBlock[32,1],
basicBlock[32,3,"Stride" -> 2,"PaddingSize"->1],
basicBlock[64,1],basicBlock[32,1],
basicBlock[32,3,"PaddingSize"->1],
basicBlock[64,1],
(* upsample layers *)
basicBlock[32,1],
DeconvolutionLayer[32,{3, 2},"Stride"->2, "PaddingSize"->1],
basicBlock[64,1],
PaddingLayer[{{0,0},{0,0},{0,1}}],
ConvolutionLayer[classNum,1],
(* classification layers *)
TransposeLayer[{1<->3,1<->2}],
SoftmaxLayer[]},
"Input"->NetEncoder[{"Image",ImageDimensions[trainData[[1,1]]],"Grayscale"}]]

GPU training is recommended, as CPU training could take up to an hour. Train the net:

results=NetTrain[net,trainData,All,
ValidationSet-> testData,
MaxTrainingRounds->9,
TargetDevice->"GPU"
]
trainedNet=results["TrainedNet"]

Create a prediction function that gives the index of the most likely class at each pixel:

pixelPredict[net_,image_]:=Map[First@Ordering[#,-1]&,Normal@net[image], {2}]

Predict the pixel classes of an image from the test set:

img=testData[[-7,1]]

ArrayPlot[pixelPredict[trainedNet,img],ColorRules->colorRules]

Plot the actual target mask of the example:

ArrayPlot[testData[[-7,2]],ColorRules->colorRules]

Obtain a sorted list of probabilities for a specific pixel to be a certain class:

prob=trainedNet[][[10,11]];
Sort[Thread[Keys@imgMap->Normal@prob],#1[[2]]>#2[[2]]&]

Define a function that rasterizes a string and then applies the trained net to give a matrix of indices corresponding to the most likely class of each pixel:

stringPredict[net_,str_]:=pixelPredict[net,ImageAssemble[Lookup[imgMap,Characters@StringPadRight[str,maxLen]]]]

Some characters are easily distinguished from other characters:

ArrayPlot[stringPredict[trainedNet,"abcdefg"],ColorRules->colorRules]

Other characters are harder:

ArrayPlot[stringPredict[trainedNet,"hqx"],ColorRules->colorRules]

Obtain the pixel-level accuracy of the trained net on the test set:

1.-Mean@Flatten[Unitize[Map[pixelPredict[trainedNet,#]&,Keys[testData]]-Values[testData]]]