Method for (RefLink[Classify, paclet:ref/Classify]).

Models class probabilities with logistic functions of linear combinations of features.

"LogisticRegression" models the log probabilities of each class with a linear combination of numerical features x={(Subscript[x, 1]),(Subscript[x, 2]),…,(Subscript[x, n])}, log(P(class = k|x))∝x.(θ^((k))), where (θ^((k)))={(Subscript[θ, 1]),(Subscript[θ, 2]),…,(Subscript[θ, m])} corresponds to the parameters for class k. The estimation of the parameter matrix θ={(θ^((1))),(θ^((2))),…,(θ^((nclass)))} is done by minimizing the loss function (∑_(i=1)^m)-log((Subscript[P, θ])(class=(Subscript[y, i])|(Subscript[x, i])))+(Subscript[λ, 1]) (∑_(i=1)^n)(Abs[(Subscript[θ, i])])+((Subscript[λ, 2])/2) (∑_(i=1)^n)((Subscript[θ, i])^2).

The following options can be given:

Possible settings for "OptimizationMethod" include:

OpenersPrimaryExamplesSection(Spacer1)Examples  (8)

OpenersExampleSection0(Spacer1)Basic Examples  (2)

Train a classifier function on labeled examples:

c=Classify[{1,2,3,4}->{1,1,2,2},Method->"LogisticRegression"]

Obtain information about the classifier:

Information[c]

Classify a new example:

c[1.3]

Generate some normally distributed data:

sampledata[center_]:=RandomVariate[MultinormalDistribution[center,IdentityMatrix[2]],200];

clusters=sampledata/@ {{1.5,1},{-1.5,1},{0,-3}};

Visualize it:

ListPlot[clusters, PlotStyle->Darker@{Yellow,Blue,Green}]

Train a classifier on this dataset:

c=Classify[<|Yellow->clusters[[1]],Blue->clusters[[2]],Green->clusters[[3]]|>, Method->"LogisticRegression"]

Plot the training set and the probability distribution of each class as a function of the features:

Show[
Plot3D[{
c[{x,y},"Probability"->Yellow],
c[{x,y},"Probability"->Blue],
c[{x,y},"Probability"-> Green]},
{x,-4,4},{y,-5,4}, 
Exclusions->None],ListPointPlot3D[Map[Append[#,1]&,clusters,{2}], PlotStyle->{Yellow,Blue,Green}]]

OpenersExampleSection1(Spacer1)Options  (6)

OpenersExampleSubsection0(Spacer1)"L1Regularization"  (2)

Train a classifier using the "L1Regularization" option:

c=Classify[<|1->{1,23,1.3, 4},2->{-4, -3.2, -4, -5}|>, Method->{"LogisticRegression","L1Regularization" -> 3}]

Generate some data and visualize it:

colors={, , ,};
clusters=Table[RandomVariate[BinormalDistribution[
RandomReal[{-3,3},2],
RandomReal[{0.5,2},2],
RandomReal[{0.2,0.8}]],RandomInteger[{30,40}]],{4}];
plot=ListPlot[clusters, PlotStyle->Darker[colors,0.1],ImageSize->200,PlotRange->{{-5,5},{-5,5}},Frame->True,AspectRatio->1,PlotLabel->"data"]

line=Range[-5,5,0.25];
points=Tuples[line,2];

makecolormap[probs_] :=Transpose @ Partition[
Map[Blend[Keys[#],Values[#]]&, probs],
Length[line]];

Train several classifiers using different values for "L1Regularization" and compare the results:

data=AssociationThread[colors,clusters];
Table[
ArrayPlot[
makecolormap @ Classify[data,points,"Probabilities",Method->{"LogisticRegression","L1Regularization"->λ}],
PlotLabel->("L1Regularization" -> λ),DataReversed->True,ImageSize->150],
{λ,{0, 3, 5, 10}}]~Multicolumn~2 ~Legended~plot

OpenersExampleSubsection1(Spacer1)"L2Regularization"  (2)

Train a classifier using the "L2Regularization" option:

c=Classify[<|1->{1,23,1.3, 4},2->{-4, -3.2, -4, -5}|>, Method->{"LogisticRegression","L2Regularization" -> 3}]

Generate some data and visualize it:

colors={, , ,};
clusters=Table[RandomVariate[BinormalDistribution[
RandomReal[{-4,3},2],
RandomReal[{0.5,2},2],
RandomReal[{0.2,0.8}]],RandomInteger[{30,40}]],{4}];
plot=ListPlot[clusters, PlotStyle->Darker[colors,0.1],ImageSize->200,PlotRange->{{-5,5},{-5,5}},Frame->True,AspectRatio->1,PlotLabel->"data"]

line=Range[-5,5,0.25];
points=Tuples[line,2];

Train several classifiers using different values for "L2Regularization" and compare the results:

makecolormap[probs_] :=Transpose @ Partition[
Map[Blend[Keys[#],Values[#]]&, probs],
Length[line]];

data=AssociationThread[colors,clusters];
Table[
ArrayPlot[
makecolormap @ Classify[data,points,"Probabilities",Method->{"LogisticRegression","L2Regularization"->λ}],
PlotLabel->("L2Regularization" -> λ),DataReversed->True,ImageSize->150],
{λ,{0, 3, 5, 10}}]~Multicolumn~2 ~Legended~plot

OpenersExampleSubsection2(Spacer1)"OptimizationMethod"  (2)

Train a classifier using a specific "OptimizationMethod":

c=Classify[<|1->{1,23,1.3, 4},2->{-4, -3.2, -4, -5}|>, Method->{"LogisticRegression","OptimizationMethod" -> "StochasticGradientDescent"}]

Train a classifier using the "Newton" method:

trainingset={->2,->5,->8,->0,->2,->7,->5,->1,->3,->0,->3,->9,->6,->2,->8,->2,->0,->6,->6,->1,->1,->7,->8,->5,->0,->4,->7,->6,->0,->2,->5,->3,->1,->5,->6,->7,->5,->4,->1,->9,->3,->6,->8,->0,->9,->3,->0,->3,->7,->4,->4,->3,->8,->0,->4,->1,->3,->7,->6,->4,->7,->2,->7,->2,->5,->2,->0,->9,->8,->9,->8,->1,->6,->4,->8,->5,->8,->0,->6,->7,->4,->5,->8,->4,->3,->1,->5,->1,->9,->9,->9,->2,->4,->7,->3,->1,->9,->2,->9,->6};

logistic=Classify[trainingset, Method->{"LogisticRegression","OptimizationMethod"->"Newton"}]

Train a classifier using the "StochasticGradientDescent" method:

logistic2=Classify[trainingset, Method->{"LogisticRegression","OptimizationMethod"->"StochasticGradientDescent"}]

Compare the corresponding training times:

ClassifierInformation[logistic, "TrainingTime"]
ClassifierInformation[logistic2, "TrainingTime"]

({{DocumentationFooterIconsRelatedFunction.png, ({{See Also}, {(RefLink[Classify, paclet:ref/Classify])  ▪  (RefLink[ClassifierFunction, paclet:ref/ClassifierFunction])  ▪  (RefLink[ClassifierMeasurements, paclet:ref/ClassifierMeasurements])  ▪  (RefLink[Predict, paclet:ref/Predict])  ▪  (RefLink[PredictorMeasurements, paclet:ref/PredictorMeasurements])  ▪  (RefLink[SequencePredict, paclet:ref/SequencePredict])  ▪  (RefLink[ClusterClassify, paclet:ref/ClusterClassify])  ▪  (RefLink[LogisticSigmoid, paclet:ref/LogisticSigmoid])  ▪    ▪    ▪    ▪    ▪    ▪  }, {Related Machine Learning Methods:  (RefLink[DecisionTree, paclet:ref/method/DecisionTree])  ▪  (RefLink[Markov, paclet:ref/method/Markov])  ▪  (RefLink[NearestNeighbors, paclet:ref/method/NearestNeighbors])  ▪  (RefLink[NeuralNetwork, paclet:ref/method/NeuralNetwork])  ▪  (RefLink[RandomForest, paclet:ref/method/RandomForest])  ▪  (RefLink[SupportVectorMachine, paclet:ref/method/SupportVectorMachine])}})}})

({{(Spacer1), ({{(Spacer1)Related Links}, {(WebLink[DocumentationFooterIconsRelatedLinks-EIWL.png(Spacer1)An Elementary Introduction to the Wolfram Language, https://www.wolfram.com/language/elementary-introduction/22-machine-learning.html])(WebLink[: Machine Learning, https://www.wolfram.com/language/elementary-introduction/22-machine-learning.html])}})}})