({{, (RefLink[BayesianMinimization, paclet:ref/BayesianMinimization])[f,{(Subscript[conf, 1]),(Subscript[conf, 2]),…}]
gives an object representing the result of Bayesian minimization of the function f over the configurations (Subscript[conf, i]). }, {, (RefLink[BayesianMinimization, paclet:ref/BayesianMinimization])[f,reg]
minimizes over the region represented by the region specification reg.}, {, (RefLink[BayesianMinimization, paclet:ref/BayesianMinimization])[f,sampler]
minimizes over configurations obtained by applying the function sampler. }, {, (RefLink[BayesianMinimization, paclet:ref/BayesianMinimization])[f,{(Subscript[conf, 1]),(Subscript[conf, 2]),…}->nsampler]
applies the function nsampler to successively generate configurations starting from the (Subscript[conf, i]). }})

OpenersNotesSection(Spacer1)Details and Options

(RefLink[BayesianMinimization, paclet:ref/BayesianMinimization])[…] returns a (RefLink[BayesianMinimizationObject, paclet:ref/BayesianMinimizationObject])[…] whose properties can be obtained using (RefLink[BayesianMinimizationObject, paclet:ref/BayesianMinimizationObject])[…]["prop"].

Possible properties include:

Configurations can be of any form accepted by (RefLink[Predict, paclet:ref/Predict]) (single data element, list of data elements, association of data elements, etc.) and of any type accepted by (RefLink[Predict, paclet:ref/Predict]) (numerical, textual, sounds, images, etc.).

The function f must output a real-number value when applied to a configuration conf.

(RefLink[BayesianMinimization, paclet:ref/BayesianMinimization])[f,…] attempts to find a good minimum using the smallest number of evaluations of f.

In (RefLink[BayesianMinimization, paclet:ref/BayesianMinimization])[f,spec], spec defines the domain of the function f. A domain can be defined by a list of configurations, a geometric region, or a configuration generator function.

In (RefLink[BayesianMinimization, paclet:ref/BayesianMinimization])[f,sampler], sampler[] must output a configuration suitable for f to be applied to it.

In (RefLink[BayesianMinimization, paclet:ref/BayesianMinimization])[f,{(Subscript[conf, 1]),(Subscript[conf, 2]),…}->nsampler], nsampler[conf] must output a configuration.

(RefLink[BayesianMinimization, paclet:ref/BayesianMinimization]) takes the following options:

Possible settings for (RefLink[Method, paclet:ref/Method]) include:

Possible settings for (RefLink[RandomSeeding, paclet:ref/RandomSeeding]) include:

OpenersPrimaryExamplesSection(Spacer1)Examples  (14)

OpenersExampleSection0(Spacer1)Basic Examples  (3)

Minimize a function over an interval:

bo=BayesianMinimization[(#-2)^2 + 1&,Interval[{0,4}]]

Use the resulting (RefLink[BayesianMinimizationObject, paclet:ref/BayesianMinimizationObject])[…] to get the estimated minimum configuration:

bo["MinimumConfiguration"]

Get the estimated minimum function value:

bo["MinimumValue"]

Minimize a function over a set of configurations:

bo=BayesianMinimization[(#^2-3*#+2)&,{5.7,1.1,3.4,6.8,2.3,1.2}]

Get the minimum configuration over the set:

bo["MinimumConfiguration"]

Minimize a function over a domain defined by a random generator:

bo=BayesianMinimization[ (#^2-5*#+4)&,RandomReal[{0,5}]&]

Get the estimated minimum value:

bo["MinimumValue"]

OpenersExampleSection1(Spacer1)Scope  (3)

Minimize a function over a region:

f[x_]:=(x-4)^2+1;
reg=Interval[{0.,7.}];

bo=BayesianMinimization[f,reg]

Get the list of available properties to query:

bo["Properties"]

Get the history of evaluations:

bo["EvaluationHistory"]

Get information about the method used to determine the configurations to explore:

bo["Method"]

Get the current probabilistic model of the function (this is a (RefLink[PredictorFunction, paclet:ref/PredictorFunction])):

p=bo["PredictorFunction"]

Find the best configuration to explore if the minimization were continued:

bo["NextConfiguration"]

Find a list of properties simultaneously:

bo[{"ObjectiveFunction", "Method", "MinimumValue"}]

Visualize how well the function is modeled, particularly near the minimum:

Plot[{f[{x}],p[{x}]}, {x, 0,7}, PlotLegends->{"function", "model"}, Exclusions-> False ]

Minimize a function with initial configurations over a domain defined by a random neighborhood configuration generator:

f[x_]:= x^2- 7*x + 3;
neighborsampler[x_]:= RandomReal[{x-0.2,x+0.2}];

bo = BayesianMinimization[f, {1.,7.,2.}-> neighborsampler]

Get the model of the function:

p = bo["PredictorFunction"]

Visualize the model's performance near the minimum:

Plot[{f[{x}],p[{x}]}, {x, 0,7}, PlotLegends->{"function", "model"} , Exclusions-> False]

Define a function that takes an image and computes the negative probability returned by (RefLink[ImageIdentify, paclet:ref/ImageIdentify]) of identifying the image as the entity (Entity["apple", Entity["Concept","Apple::857h7"], "Entity[\"Concept\", \"Apple::857h7\"]", "concept"]), with the domain defined by a random generator over a corpus of images:

f[image_]:=Module[{negprob}, negprob = - ImageIdentify[image, (Entity["fruit", Entity["Concept","Fruit::855t9"], "Entity[\"Concept\", \"Fruit::855t9\"]", "concept"]), 5, "Probability"];
	Lookup[negprob, (Entity["apple", Entity["Concept","Apple::857h7"], "Entity[\"Concept\", \"Apple::857h7\"]", "concept"]),0.]
];
images=WikipediaData["Apple", "ImageList"];
sampler[]:= RandomChoice[images];

Minimize the function above:

bo = BayesianMinimization[f, sampler, AssumeDeterministic-> True]

Get the minimum configuration:

bo["MinimumConfiguration"]

Get the evaluation history:

bo["EvaluationHistory"]

OpenersExampleSection2(Spacer1)Options  (4)

AssumeDeterministic  (1)

Minimize a function over a domain defined by a random generator:

f[x_]:=-x Sin[x]+3 Cos[x]+x/2;
sampler[]:= RandomReal[{-6., 6.}];

bo=BayesianMinimization[f,sampler]

The function is assumed to be stochastic; the value from the probabilistic model will differ in general from the function value for evaluated configurations:

bo["MinimumValue"]

f[bo["MinimumConfiguration"]]

Include information that the function is deterministic, i.e. noise free:

bo2 = BayesianMinimization[f,sampler, AssumeDeterministic -> True]

For a deterministic function, the model value and the function value for evaluated configurations agree to a good precision:

bo2["MinimumValue"]

f[bo2["MinimumConfiguration"]]

OpenersExampleSubsection0(Spacer1)MaxIterations  (1)

Minimize a function with a domain defined by a random generator:

f[x_]:= Sin[x]+4*Cos[x]+x;
sampler []:= RandomReal[{-5,5}];

bo= BayesianMinimization[f,sampler]

Get the number of function evaluations:

Length @ bo["EvaluationHistory"]

Specify the maximum number of iterations:

bo2 = BayesianMinimization[f,sampler, MaxIterations -> 5]

Length @ bo2["EvaluationHistory"]

OpenersExampleSubsection1(Spacer1)Method  (1)

Define a function over an interval region:

f[x_]:=  x^2-3*x+2;
int = Interval[{-5.,5.}];

Specify the method for exploring configurations:

bo= BayesianMinimization[f,int, Method -> "MaxExpectedImprovement"]

Specify a different method:

bo2= BayesianMinimization[f,int, Method -> "MaxImprovementProbability"]

OpenersExampleSubsection2(Spacer1)InitialEvaluationHistory  (1)

Minimize a function over a disk region with a small number of iterations:

f[{x_, y_}]:=  Cos[x]-Exp[(x-0.5) *y];
reg = Disk[{0,0}, 1.];

bo= BayesianMinimization[f,reg, MaxIterations-> 5]

Use the information from this evaluation in the next:

evalhist = bo["EvaluationHistory"]

bo2= BayesianMinimization[f,reg, InitialEvaluationHistory-> evalhist]

Get the estimated minimum configuration now:

bo2["MinimumConfiguration"]

OpenersExampleSection3(Spacer1)Applications  (2)

Define a training set to train predictor functions using the (RefLink[Predict, paclet:ref/Predict]) function and a test set to measure their performance:

trainingset = Table[n-> Sin[n],{n,1,15}];
testset = {3.9-> Sin[3.9], 7.3-> Sin[7.3]};

Create a "loss" function to test the performance of different methods in (RefLink[Predict, paclet:ref/Predict]):

func[method_]:= -PredictorMeasurements[Predict[trainingset,Method-> method], testset, "LogLikelihood"];
methods= {"LinearRegression", "NearestNeighbors", "NeuralNetwork", "GaussianProcess"};

Minimize the loss function over a domain defined by a list of different methods for (RefLink[Predict, paclet:ref/Predict]):

bo=BayesianMinimization[func, methods]

Examine the evaluation history:

bo["EvaluationHistory"]

Find the best configuration to explore next:

bo["NextConfiguration"]

Load Fisher's Iris dataset and divide it into a training set and a validation set:

sampletest =  RandomSample[ExampleData[{"MachineLearning", "FisherIris"}, "Data"]];
trainingsample= sampletest[[1;;100]];
validationsample = sampletest[[101;;]];

Create "blackbox" functions. Here the functions are the loss (negative log-likelihood) functions for two different methods used in the (RefLink[Classify, paclet:ref/Classify]) function. The arguments of the functions are known as hyperparameters.

Train a logistic regression classifier with two hyperparameters, the L1 and L2 regularization coefficients:

lossLogisticReg[{l1_, l2_}]:= Module[
	{class},
	class = Classify[trainingsample, Method -> {"LogisticRegression", 
		"L1Regularization"-> Exp[l1], "L2Regularization"-> Exp[l2]}];
	-ClassifierMeasurements[class, validationsample, "LogLikelihood"]
];
reg = Rectangle[{-3.,-3.},{3.,3.}];

Minimize the loss function for the logistic regression classifier over a domain defined by a rectangular region in the logarithm of the hyperparameters:

boLogistic= BayesianMinimization[lossLogisticReg, reg]

Get the model of the function:

pLogistic = boLogistic["PredictorFunction"]

Visualize the model of the loss function of the classifier together with the estimated minimum:

pt = {Append[boLogistic["MinimumConfiguration"], boLogistic["MinimumValue"]]}

Show[Plot3D[pLogistic[{x,y}],{x,-3,3},{y,-3,3},PlotLegends->{"Logistic Regression"}, Exclusions-> False], ListPointPlot3D[pt, PlotStyle-> Directive[Red, PointSize[Large]]]]

Now train a support vector machine (SVM) classifier with two hyperparameters, the soft margin parameter and the gamma scaling parameter:

lossSVMRBF[{c_, gamma_}]:= Module[
	{class},
	class =  Classify[trainingsample, Method -> {"SupportVectorMachine", "KernelType"-> 
		"RadialBasisFunction", "SoftMarginParameter"-> Exp[c], "GammaScalingParameter"-> Exp[gamma]}];
	-ClassifierMeasurements[class, validationsample, "LogLikelihood"]
];
reg2 = Rectangle[{-3.,-3.},{3.,3.}];

Minimize the loss function for the SVM classifier over a domain defined by a rectangular region in the logarithm of the hyperparameters:

boSVM= BayesianMinimization[lossSVMRBF, reg2]

Get the model of the function:

pSVM = boSVM["PredictorFunction"]

Visualize the model of the loss function together with the estimated minimum:

pt2= {Append[boSVM["MinimumConfiguration"], boSVM["MinimumValue"]]}

Show[Plot3D[pSVM[{x,y}],{x,-3,3},{y,-3,3}, ColorFunction-> "BlueGreenYellow",PlotLegends->"Support Vector Machine-RBF", Exclusions-> False], ListPointPlot3D[pt2, PlotStyle-> Directive[Red, PointSize[Large]]]]

OpenersExampleSection4(Spacer1)Possible Issues  (2)

When the domain of the objective function is defined by an initial configuration set and a neighborhood configuration generator, the results depend on the quality of the generator provided.

Minimize a function where the domain is defined as above:

f[x_]:= Sin[x]-x*Cos[x];
ns[x_]:= RandomReal[{x-0.1,x+0.1}];

bo = BayesianMinimization[f, {0,-3.}-> ns, Method-> Automatic, AssumeDeterministic-> True]

Get the model of the function:

p = bo["PredictorFunction"]

Since the starting initial configurations are "far" from the global minimum and the generator takes relatively small steps, the algorithm could converge to a local minimum:

Plot[{f[x], p[x]},{x,-8, 8}, Exclusions-> False]

If the neighborhood configuration generator takes steps that are "too large" or "too small", this could lead to problems:

bo=BayesianMinimization[Norm,Range[2,5]->(#^2&),AssumeDeterministic->True]

In this case, at each step the generator gives a configuration value that is square of the previous one, so the values quickly become extremely large and the probabilistic model does not work properly.

({{DocumentationFooterIconsRelatedFunction.png, ({{See Also}, {(RefLink[BayesianMaximization, paclet:ref/BayesianMaximization])  ▪  (RefLink[BayesianMinimizationObject, paclet:ref/BayesianMinimizationObject])  ▪  (RefLink[NMinimize, paclet:ref/NMinimize])  ▪  (RefLink[FindMinimum, paclet:ref/FindMinimum])  ▪  (RefLink[Predict, paclet:ref/Predict])}})}})

({{DocumentationFooterIconsRelatedGuide.png, ({{Related Guides}, {▪(RefLinkPlain[Machine Learning, paclet:guide/MachineLearning])}, {▪(RefLinkPlain[Optimization, paclet:guide/Optimization])}})}})