Plotting Search Data

The utility functions FindMinimumPlot and FindRootPlot show search data for (RefLink[FindMinimum, paclet:ref/FindMinimum]) and (RefLink[FindRoot, paclet:ref/FindRoot]) for one- and two-dimensional functions. They work with essentially the same arguments as (RefLink[FindMinimum, paclet:ref/FindMinimum]) and (RefLink[FindRoot, paclet:ref/FindRoot]) except that they additionally take options, which affect the graphics functions they call to provide the plots, and they do not have the (RefLink[HoldAll, paclet:ref/HoldAll]) attribute as do (RefLink[FindMinimum, paclet:ref/FindMinimum]) and (RefLink[FindRoot, paclet:ref/FindRoot]).

Plotting search data.

Note that to simplify processing and reduce possible confusion about the function f, FindRootPlot does not accept equations; it finds a root f=0.

Steps and evaluation points are color coded for easy detection as follows:

FindMinimumPlot and FindRootPlot return a list containing {result,summary,plot}, where:

This loads the package:

<<Optimization`UnconstrainedProblems`

This shows in two dimensions the steps and evaluations used by (RefLink[FindMinimum, paclet:ref/FindMinimum]) to find a local minimum of the function cos((x^2)-3 y)+sin((x^2)+(y^2)) starting at the point {x,y}=={1,1}. Options are given to (RefLink[ContourPlot, paclet:ref/ContourPlot]) so that no contour lines are shown and the function value is indicated by grayscale. Since (RefLink[FindMinimum, paclet:ref/FindMinimum]) by default uses the quasi-Newton method, there are only evaluations of the function and gradient that occur at the same points, indicated by the red circles with green centers:

FindMinimumPlot[Cos[x^2 - 3 y] + Sin[x^2 + y^2],{{x,1},{y,1}},Contours->100, ContourLines->False]

This shows in two dimensions the steps and evaluations used by (RefLink[FindMinimum, paclet:ref/FindMinimum]) to find a local minimum of the function ((((x^2)-3y))^2)+(sin^2)((x^2)+(y^2)) starting at the point {x,y}=={1,1}. Since the problem is a sum of squares, (RefLink[FindMinimum, paclet:ref/FindMinimum]) by default uses the Gauss–Newton/Levenberg–Marquardt method that derives a residual function and only evaluates it and its Jacobian. Points at which the residual function is evaluated are shown with yellow dots. The yellow dots surrounded by a large purple circle are points at which the Jacobian was evaluated as well:

FindMinimumPlot[(x^2 - 3 y)^2 + Sin[x^2 + y^2]^2,{{x,1},{y,1}}]

This shows in two dimensions the steps and evaluations used by (RefLink[FindMinimum, paclet:ref/FindMinimum]) to find a local minimum of the function ((((x^2)-3y))^2)+(sin^2)((x^2)+(y^2)) starting at the point {x,y}=={1,1} using Newton's method. Points at which the function, gradient, and Hessian were all evaluated are shown by concentric green, red, and cyan circles. Note that in this example, all the Newton steps satisfied the Wolfe conditions, so there were no points where the function and gradient were evaluated separately from the Hessian, which is not always the case. Note also that Newton's method finds a different local minimum than the default method:

FindMinimumPlot[(x^2 - 3 y)^2 + Sin[x^2 + y^2]^2,{{x,1},{y,1}}, Method->Newton]

This shows the steps and evaluations used by (RefLink[FindMinimum, paclet:ref/FindMinimum]) to find a local minimum of the function (e^x)+(1/x) with two starting values superimposed on the plot of the function. Options are given to (RefLink[Plot, paclet:ref/Plot]) so that the curve representing the function is thick and purple. With two starting values, (RefLink[FindMinimum, paclet:ref/FindMinimum]) uses the derivative-free principal axis method, so there are only function evaluations, indicated by the green dots:

FindMinimumPlot[Exp[x] + 1/x,{x,1, 1.1}, PlotStyle->{Thickness[.025], RGBColor[.4,0,.4]}]

This shows in two dimensions the steps and evaluations used by (RefLink[FindRoot, paclet:ref/FindRoot]) to find a root of the function {(x^2)-3y,sin((x^2)+(y^2))}=={0,0} starting at the point {x,y}=={1,1}. As described earlier, the function is a residual, and the default method in (RefLink[FindRoot, paclet:ref/FindRoot]) evaluates the residual and its Jacobian as shown by the yellow dots and purple circles. Note that this plot is nearly the same as the one produced by FindMinimumPlot with the default method for the function ((((x^2)-3y))^2)+(sin^2)((x^2)+(y^2)) since the residual is the same. FindRootPlot also shows the zero contour of each component of the residual function in red and green:

FindRootPlot[{x^2 - 3 y,Sin[x^2 + y^2]},{{x,1},{y,1}}]