Method for (RefLink[Classify, paclet:ref/Classify]) and (RefLink[Predict, paclet:ref/Predict]).

Predict the value or class of an example using an ensemble of decision trees.

Trees are trained sequentially following the boosting meta-algorithm.

Gradient boosting is a machine learning technique for regression and classification problems that produces a prediction model in the form of an ensemble of trees. Trees are trained sequentially with the goal of compensating the weaknesses of previous trees. The current implementation uses the LightGBM framework in the back end.

The following options can be given:

Possible settings for "BoostingMethod" include "Gradient", "GradientOneSideSampling", and "DART" (i.e. Dropouts meet Multiple Additive Regression Trees).

OpenersPrimaryExamplesSection(Spacer1)Examples  (10)

OpenersExampleSection0(Spacer1)Basic Examples  (2)

Train a predictor function on labeled examples:

p=Predict[{1,2,3,4}->{.3,.4,.6,9},Method->"GradientBoostedTrees"]

Obtain information about the predictor:

Information[p]

Predict a new example:

p[1.3]

Generate some data and visualize it:

data=Table[x->x^2+RandomVariate[NormalDistribution[0, 5]], {x, RandomReal[{-10, 10}, 200]}];
ListPlot[List@@@data]

Train a predictor function on it:

p=Predict[data,Method->"GradientBoostedTrees"]

Compare the data with the predicted values and look at the standard deviation:

Show[Plot[
{p[x],
p[x]+StandardDeviation[p[x,"Distribution"]],p[x]-StandardDeviation[p[x,"Distribution"]]},
{x,-2,6},
PlotStyle->{Blue,Gray,Gray},
Filling->{2->{3}},
Exclusions->False,
PerformanceGoal->"Speed",PlotLegends->{"Prediction","Confidence Interval"}],ListPlot[List@@@data,PlotStyle->Red,PlotLegends->{"Data"}]]

OpenersExampleSection1(Spacer1)Options  (8)

OpenersExampleSubsection0(Spacer1)MaxTrainingRounds  (2)

Use the (RefLink[MaxTrainingRounds, paclet:ref/MaxTrainingRounds]) option to train a classifier:

c = Classify[{1,2,3,4,5, 6}->{1,1,3,3,1,3}, Method->{"GradientBoostedTrees", MaxTrainingRounds->5}]

Train two classifiers on the "Mushroom" dataset by changing the value of (RefLink[MaxTrainingRounds, paclet:ref/MaxTrainingRounds]):

trainingset=ExampleData[{"MachineLearning","Mushroom"},"TrainingData"];

c3=Classify[trainingset, Method->{"GradientBoostedTrees",MaxTrainingRounds->3}]

c10=Classify[trainingset, Method->{"GradientBoostedTrees",MaxTrainingRounds->10}]

Look at how the performance increases:

testset=ExampleData[{"MachineLearning","Mushroom"},"TestData"];

ClassifierMeasurements[c3,testset,"Accuracy"]

ClassifierMeasurements[c10,testset,"Accuracy"]

OpenersExampleSubsection1(Spacer1)"BoostingMethod"  (1)

Train two classifiers on the "WineQuality" training set using a different "BoostingMethod" for each, and compare the training time:

trainingdata=ExampleData[{"MachineLearning", "WineQuality"}, "TrainingData"];

{{t1, c1},{t2, c2}} = AbsoluteTiming[Classify[trainingdata, Method->{"GradientBoostedTrees","BoostingMethod"->#}]]&/@{"Gradient","DART"}

Compare the accuracy on a test set:

testdata=ExampleData[{"MachineLearning", "WineQuality"}, "TestData"];

ClassifierMeasurements[#,testdata,"Accuracy"]&/@{c1, c2}

OpenersExampleSubsection2(Spacer1)"LeafSize"  (2)

Train a predictor function using the "LeafSize" option:

Predict[{1,2,3,-2,-4, -5}->{.1, .2, .6, 1.3, 6.7, 8},
Method->{"GradientBoostedTrees","LeafSize"->1}]

Train two classifiers on the "Titanic" dataset by changing the value of "LeafSize":

trainingset=ExampleData[{"MachineLearning","Titanic"},"TrainingData"];

c3=Classify[trainingset, Method->{"GradientBoostedTrees","LeafSize"->3}]

c400=Classify[trainingset, Method->{"GradientBoostedTrees","LeafSize"->400}]

Look at how the performance increases:

testset=ExampleData[{"MachineLearning","Titanic"},"TestData"];

ClassifierMeasurements[#,testset,"Accuracy"]&/@{c3,c400}

OpenersExampleSubsection3(Spacer1)"LeavesNumber"  (1)

Generate a labeled training set:

trainingset=Table[x->Cos[x]+RandomVariate[NormalDistribution[0,.5]], {x,RandomReal[{-20, 20},300]}];
ListPlot[List@@@trainingset]

Train two predictors using a different "LeavesNumber" for each:

p5=Predict[trainingset, Method->{"GradientBoostedTrees","LeavesNumber"->5}
]

p80=Predict[trainingset, Method->{"GradientBoostedTrees","LeavesNumber"->80}
]

Compare the performance:

Show[ListPlot[List@@@trainingset],Plot[{p5[x],p80[x]},{x,-20,20}]]

OpenersExampleSubsection4(Spacer1)"MaxDepth"  (2)

Use the "MaxDepth" option to train a classifier:

c=Classify[{1,2,3,4,5, 6}->{1,1,3,3,1,3}, Method->{"GradientBoostedTrees","MaxDepth"->3}]

Use the "BostonHomes" training set to train two predictors with a different "MaxDepth" for each:

trainingdata=ExampleData[{"MachineLearning","BostonHomes"}, "TrainingData"];

p2=Predict[trainingdata, Method->{"GradientBoostedTrees", "MaxDepth"->2}
]

p20=Predict[trainingdata, Method->{"GradientBoostedTrees", "MaxDepth"->20}
]

Compare the "ComparisonPlot" on a test set:

testdata=ExampleData[{"MachineLearning","BostonHomes"}, "TestData"];

Grid[{PredictorMeasurements[#,testdata,"ComparisonPlot"]&/@{p2, p20}},Frame->All]

({{DocumentationFooterIconsRelatedFunction.png, ({{See Also}, {(RefLink[Classify, paclet:ref/Classify])  ▪  (RefLink[Predict, paclet:ref/Predict])  ▪  (RefLink[ClassifierFunction, paclet:ref/ClassifierFunction])  ▪  (RefLink[PredictorFunction, paclet:ref/PredictorFunction])  ▪  (RefLink[ClassifierMeasurements, paclet:ref/ClassifierMeasurements])  ▪  (RefLink[PredictorMeasurements, paclet:ref/PredictorMeasurements])  ▪  (RefLink[SequencePredict, paclet:ref/SequencePredict])  ▪  (RefLink[ClusterClassify, paclet:ref/ClusterClassify])  ▪    ▪    ▪    ▪    ▪    ▪    ▪    ▪    ▪    ▪  }, {Related Machine Learning Methods:  (RefLink[DecisionTree, paclet:ref/method/DecisionTree])  ▪  (RefLink[LinearRegression, paclet:ref/method/LinearRegression])  ▪  (RefLink[LogisticRegression, paclet:ref/method/LogisticRegression])  ▪  (RefLink[GaussianProcess, paclet:ref/method/GaussianProcess])  ▪  (RefLink[Markov, paclet:ref/method/Markov])  ▪  (RefLink[NaiveBayes, paclet:ref/method/NaiveBayes])  ▪  (RefLink[NearestNeighbors, paclet:ref/method/NearestNeighbors])  ▪  (RefLink[NeuralNetwork, paclet:ref/method/NeuralNetwork])  ▪  (RefLink[RandomForest, paclet:ref/method/RandomForest])  ▪  (RefLink[SupportVectorMachine, paclet:ref/method/SupportVectorMachine])}})}})

({{(Spacer1), ({{(Spacer1)Related Links}, {(WebLink[DocumentationFooterIconsRelatedLinks-EIWL.png(Spacer1)An Elementary Introduction to the Wolfram Language, https://www.wolfram.com/language/elementary-introduction/22-machine-learning.html])(WebLink[: Machine Learning, https://www.wolfram.com/language/elementary-introduction/22-machine-learning.html])}})}})