Method for (RefLink[Predict, paclet:ref/Predict]).

Predict values using a linear combination of features.

The linear regression predicts the numerical output y using a linear combination of numerical features x={(Subscript[x, 1]),(Subscript[x, 2]),…,(Subscript[x, n])}. The conditional probability P(y|x) is modeled according to P(y|x)∝exp(-(((y-f(θ,x)))^2)/(2 (σ^2))), with f(θ,x)=x.θ.

The estimation of the parameter vector θ is done by minimizing the loss function (1/2)(∑_(i=1)^m)((((Subscript[y, i])-f(θ,(Subscript[x, i]))))^2)+(Subscript[λ, 1])(∑_(i=1)^n)(Abs[(Subscript[θ, i])])+((Subscript[λ, 2])/2) (∑_(i=1)^n)((Subscript[θ, i])^2), where m is the number of examples and n is the number of numerical features.

The following suboptions can be given:

Possible settings for the "OptimizationMethod" option include:

For this method, (RefLink[PredictorInformation, paclet:ref/PredictorInformation])[(RefLink[PredictorFunction, paclet:ref/PredictorFunction])[…],"Function"] gives a simple expression to compute the predicted value from the features.

OpenersPrimaryExamplesSection(Spacer1)Examples  (7)

OpenersExampleSection0(Spacer1)Basic Examples  (2)

Train a predictor on labeled examples:

p=Predict[{1,2,3,4}->{.3,.4,.6,9},Method->"LinearRegression"]

Look at the (RefLink[PredictorInformation, paclet:ref/PredictorInformation]):

PredictorInformation[p]

Predict a new example:

p[1.3]

Generate two-dimensional data:

data=Table[x->x+RandomVariate[NormalDistribution[]], {x,RandomReal[{-5,5},40]}];
ListPlot[List@@@data]

Train a predictor function on it:

p=Predict[data,Method->"LinearRegression"]

Compare the data with the predicted values and look at the standard deviation:

Show[Plot[{
p[x],
p[x]+StandardDeviation[p[x,"Distribution"]],p[x]-StandardDeviation[p[x,"Distribution"]]},
{x,-2,6},
PlotStyle->{Blue,Gray,Gray},
Filling->{2->{3}},
Exclusions->False,
PerformanceGoal->"Speed",PlotLegends->{"Prediction","Confidence Interval"}
],
ListPlot[List@@@data,PlotStyle->Red,PlotLegends->{"Data"}]
]

OpenersExampleSection1(Spacer1)Options  (5)

OpenersExampleSubsection0(Spacer1)"L1Regularization"  (2)

Use the "L1Regularization" option to train a predictor:

p=Predict[{1,2,3,4}->{.3,.4,.6,9},Method->{"LinearRegression","L1Regularization"-> 1}]

Generate a training set and visualize it:

trainingset = Flatten[Table[{x, y}-> x+RandomReal[1],{x,RandomReal[{-5, 5}, 20]}, {y, RandomReal[{-5, 5}, 20]}], 1];
ListPointPlot3D[Flatten/@List@@@trainingset]

Train two predictors by using different values of the "L1Regularization" option:

p0=Predict[trainingset,Method->{"LinearRegression","L1Regularization"->0}]

p7=Predict[trainingset,Method->{"LinearRegression","L1Regularization"->7}]

Look at the predictor function to see how the larger L1 regularization has forced one parameter to be zero:

PredictorInformation[p0, "Function"]

PredictorInformation[p7, "Function"]

OpenersExampleSubsection1(Spacer1)"L2Regularization"  (2)

Use the "L2Regularization" option to train a predictor:

p=Predict[{1,2,3,4}->{.3,.4,.6,9},Method->{"LinearRegression","L2Regularization"-> 1}]

Generate a training set and visualize it:

trainingset=Table[x->x+RandomReal[{-3,3}], {x,RandomReal[{-5,5},20]}];
ListPlot[List@@@trainingset]

Train two predictors by using different values of the "L2Regularization" option:

p0=Predict[trainingset,Method->{"LinearRegression","L2Regularization"->0}]

p5=Predict[trainingset,Method->{"LinearRegression","L2Regularization"->5}]

Look at the predictor functions to see how the L2 regularization has reduced the norm of the parameter vector:

{f1,f2}=PredictorInformation[#,"Function"]&/@{p0,p5}

{theta1,theta2}=Most[Cases[#,_?NumericQ,Infinity]]&/@{f1,f2};
Norm/@{theta1,theta2}

OpenersExampleSubsection2(Spacer1)"OptimizationMethod"  (1)

Generate a large training set:

n=20000;
dim=20;
trainingset=RandomReal[{-5,5},{n,dim}]->RandomReal[1,n];

Train predictors with different optimization methods and compare their training times:

AbsoluteTiming[p1=Predict[trainingset, Method->{"LinearRegression", "OptimizationMethod"->"NormalEquation"}];]

AbsoluteTiming[p2=Predict[trainingset, Method->{"LinearRegression", "OptimizationMethod"->"OrthantWiseQuasiNewton"}];]

({{DocumentationFooterIconsRelatedFunction.png, ({{See Also}, {(RefLink[Predict, paclet:ref/Predict])  ▪  (RefLink[PredictorFunction, paclet:ref/PredictorFunction])  ▪  (RefLink[LinearModelFit, paclet:ref/LinearModelFit])  ▪  (RefLink[Fit, paclet:ref/Fit])  ▪  (RefLink[LeastSquares, paclet:ref/LeastSquares])  ▪  (RefLink[GeneralizedLinearModelFit, paclet:ref/GeneralizedLinearModelFit])}, {Related Machine Learning Methods:  (RefLink[DecisionTree, paclet:ref/method/DecisionTree])  ▪  (RefLink[GaussianProcess, paclet:ref/method/GaussianProcess])  ▪  (RefLink[GradientBoostedTrees, paclet:ref/method/GradientBoostedTrees])  ▪  (RefLink[NearestNeighbors, paclet:ref/method/NearestNeighbors])  ▪  (RefLink[NeuralNetwork, paclet:ref/method/NeuralNetwork])  ▪  (RefLink[RandomForest, paclet:ref/method/RandomForest])}})}})

({{(Spacer1), ({{(Spacer1)Related Links}, {(WebLink[DocumentationFooterIconsRelatedLinks-EIWL.png(Spacer1)An Elementary Introduction to the Wolfram Language, https://www.wolfram.com/language/elementary-introduction/22-machine-learning.html])(WebLink[: Machine Learning, https://www.wolfram.com/language/elementary-introduction/22-machine-learning.html])}})}})