Classifying Data with Neural Networks

Classifying Vector Data

Create an artificial dataset from three normally distributed clusters:

makeCluster[class_,μ_,ρ_]:=RandomVariate[MultinormalDistribution[μ,{{1,2ρ},{2ρ,4}}],600]->class;

clusters=makeCluster@@@{{Red,{1.5,1.5},-.2},{Green,{-1.5,1},.1},{Blue,{0,-2.5},.8}};

Plot the dataset:

plot=ListPlot[Keys[clusters], PlotStyle->{,,},AspectRatio->1,Frame->True,PlotRange->{{-5,5},{-5,5}}]

The training data consists of rules mapping the point to the cluster it is in:

trainingData=Flatten[Thread/@clusters];
RandomSample[trainingData,6]

Create a net to compute the probability of a point lying in each cluster, using a "Class" decoder to classify the input as (RefLink[Red, paclet:ref/Red]), (RefLink[Green, paclet:ref/Green]) or (RefLink[Blue, paclet:ref/Blue]):

net=NetChain[{LinearLayer[30],ElementwiseLayer[Ramp],LinearLayer[3],SoftmaxLayer[]},"Input"->{2},"Output"->NetDecoder[{"Class",{Red,Green,Blue}}]]

Train the net on the data:

results=NetTrain[net,trainingData,All]
trained=results["TrainedNet"]

Evaluate the net on the centers of each cluster:

trained[{{1.5,1}, {-1.5,1},{0,-3}}]

Show the contours in feature space in which each class reaches a posterior probability of 0.75:

Show[plot,Quiet@Table[ContourPlot[trained[{x,y},{"Probability",color}]==0.75,{x,-5,5},{y,-5,5},ContourStyle->{Thick,Darker[color,.3]}],{color,{Red,Green,Blue}}]]

Plot the degree of uncertainty of the classifier as measured by entropy as a function of the position:

ArrayPlot[Table[trained[{x,y},"Entropy"],{y,5,-4,-.1},{x,-4,4,.1}]]

Perform logistic regression on the Fisher Iris dataset. First, obtain the training data:

trainingData=ExampleData[{"MachineLearning","FisherIris"},"TrainingData"];
testData=ExampleData[{"MachineLearning","FisherIris"},"TestData"];

RandomSample[trainingData,5]

Build a list of the unique labels to which each example is assigned:

labels=Union[Values[trainingData]]

Create a (RefLink[NetChain, paclet:ref/NetChain]) to perform the classification, using a "Class" decoder to interpret the output of the net as probabilities for each class:

net=NetChain[{3,SoftmaxLayer[]},"Input"->4,"Output"->NetDecoder[{"Class",labels}]]

(RefLink[NetTrain, paclet:ref/NetTrain]) will automatically use a (RefLink[CrossEntropyLossLayer, paclet:ref/CrossEntropyLossLayer]) with a (RefLink[NetEncoder, paclet:ref/NetEncoder]) to interpret the class labels in the training data:

results=NetTrain[net,trainingData,All]
trained=results["TrainedNet"]

Classify an input:

trained[{6.1`,2.8`,4.`,1.3`}]

Obtain the probabilities associated with the classification:

trained[{6.1`,2.8`,4.`,1.3`},"Probabilities"]

Use (RefLink[NetMeasurements, paclet:ref/NetMeasurements]) to test the classification performance of the trained net on the test set:

NetMeasurements[trained,testData,"Accuracy"]

NetMeasurements[trained,testData,"ConfusionMatrixPlot"]

Classifying Categorical Data

Perform logistic regression on a dataset containing both categorical and numeric values.

First, obtain the training data, and drop rows containing missing data:

data=ExampleData[{"Dataset","Titanic"}];
data=DeleteMissing[data,1,2]

Split the data into training and test datasets:

{trainingData,testData}=TakeDrop[RandomSample[data],800];

Categorical variables cannot be used directly in neural networks and must be encoded as arrays.

Create "Class" encoders that encode the categorical variables as one-hot encoded vectors:

classEncoder=NetEncoder[{"Class",{"1st","2nd","3rd"}, "UnitVector"}]
genderEncoder=NetEncoder[{"Class",{"male","female"},"UnitVector"}]

Applying the encoders to class labels produces unit vectors:

classEncoder/@{"1st","2nd","3rd"}

genderEncoder/@{"male","female"}

Create a network with an input corresponding to each feature and using a "Boolean" decoder to interpret the output of the net as the probability of survival.

The input features are first concatenated together before being further processed:

net=NetGraph[{CatenateLayer[],LinearLayer[],LogisticSigmoid},{{NetPort["class"],NetPort["age"],NetPort["sex"]}->1,1->2->3->NetPort["survived"]},"class"-> classEncoder,"age"->"Scalar","sex"-> genderEncoder,"survived"->"Boolean"]

Train the net on the training data. (RefLink[NetTrain, paclet:ref/NetTrain]) will automatically attach a (RefLink[CrossEntropyLossLayer, paclet:ref/CrossEntropyLossLayer])["Binary"] layer to the output of the net:

results=NetTrain[net,trainingData,All,MaxTrainingRounds->1000]
trained=results["TrainedNet"]

Predict whether a passenger will survive:

trained[<|"class"-> "2nd","age"->24,"sex"->"female"|>]

Obtain the probability of survival associated with the input:

trained[<|"class"-> "2nd","age"->24,"sex"->"female"|>,None]

Plot the survival probability as a function of age for some combinations of "class" and "sex":

p[class_,age_,sex_]:=trained[<|"class"->class,"age"->age,"sex"->sex|>,None];

Plot[{p["1st",x,"female"],p["3rd",x,"female"],p["1st",x,"male"],p["3rd",x,"male"]}, {x,0,100},PlotLegends->{"female, 1st class", "female, 3rd class", "male, 1st class", "male, 3rd class"}, Frame->True,FrameLabel-> {"age (years)", "survival probability"}]

Use (RefLink[NetMeasurements, paclet:ref/NetMeasurements]) to test the accuracy of the trained net on the test set:

NetMeasurements[trained,testData,"Accuracy"]

The accuracy is typically comparable to that obtained using (RefLink[Classify, paclet:ref/Classify]) when specifying the method "LogisticRegression":

cf=Classify[trainingData->"survived",Method-> "LogisticRegression"];
ClassifierMeasurements[cf,testData->"survived","Accuracy"]

Multitask Classification

Perform multitask learning by creating a net that produces two separate classifications.

First, obtain training data:

obj=ResourceObject["CIFAR-100"]

The training data consists of an image and the corresponding high-level and low-level labels:

trainingData=ResourceData[obj,"TrainingDataset"];
RandomSample[trainingData,5]

Extract the unique labels from the "Label" and "SubLabel" columns:

labels=Union@Normal@trainingData[All,"Label"]

sublabels=Union@Normal@trainingData[All,"SubLabel"]

Create a base convolutional net that will produce a vector of 500 features:

convnet=NetChain[{
ConvolutionLayer[20,{5,5}],
ElementwiseLayer[Ramp],
PoolingLayer[{2,2},{2,2}],
ConvolutionLayer[50,{5,5}],
ElementwiseLayer[Ramp],
PoolingLayer[{2,2},{2,2}],
FlattenLayer[],
LinearLayer[500],
ElementwiseLayer[Ramp]
},"Input"->NetEncoder[{"Image",{32,32}}]]

Create a (RefLink[NetGraph, paclet:ref/NetGraph]) that will produce separate classifications for the high-level and low-level labels:

net=NetGraph[{convnet,100,SoftmaxLayer[],20,SoftmaxLayer[]},{NetPort["Image"]->1->2->3->NetPort["SubLabel"],2->4->5->NetPort["Label"]},
"Label"->NetDecoder[{"Class",labels}],"SubLabel"->NetDecoder[{"Class",sublabels}]]

Train the network. (RefLink[NetTrain, paclet:ref/NetTrain]) will automatically attach (RefLink[CrossEntropyLossLayer, paclet:ref/CrossEntropyLossLayer]) objects to both outputs, taking the target values from the training data using the corresponding names "Label" and "SubLabel":

results=NetTrain[net,trainingData,All,MaxTrainingRounds->10,TargetDevice->"GPU"]
trained=results["TrainedNet"]

Evaluate the trained network on a single image:

trained[]

Evaluate the trained network on several images, taking only the "SubLabel" output:

trained[{,,,,},"SubLabel"]

Get a property of the class decoder of the "SubLabel" output on an input image:

trained[,"SubLabel"->{"TopProbabilities",3}]

From a random sample, select the images for which the net produces highest and lowest entropy predictions for "Label":

images=RandomSample[Normal[trainingData][[All,"Image"]],5000];

entropies=trained[images,"Label"->"Entropy"];

Labeled[images[[Ordering[entropies,-10]]],"high entropy"]
Labeled[images[[Ordering[entropies,10]]],"low entropy"]

Use (RefLink[NetMeasurements, paclet:ref/NetMeasurements]) to test the accuracy for both outputs of the net:

NetMeasurements[trained,trainingData,{<|"Measurement"->"Accuracy","Source"->"Label","Key"->"Acc1"|>,<|"Measurement"->"Accuracy","Source"->"SubLabel","Key"->"Acc2"|>}]

Produce a subnetwork that computes only "SubLabel" predictions:

subnet=Take[trained,{NetPort["Image"],NetPort["SubLabel"]}]

Make a prediction on a single image:

subnet[]