Documentation

Parameter Estimation

One approach for estimating the parameters $$\beta$$, $$\theta$$, and $${\sigma }^2$$ of a GPR model is by maximizing the likelihood $$P\left(y|X\right)$$ as a function of $$\beta$$, $$\theta$$, and $${\sigma }^2$$[1]. That is, if $$\widehat{\beta }$$, $$\widehat{\theta }$$, and $${\widehat{\sigma }}^2$$ are the estimates of $$\beta$$, $$\theta$$, and $${\sigma }^2$$, respectively, then:

Because

the marginal log likelihood function is as follows:

where $$H$$ is the vector of explicit basis functions, and $$K\left(X,X|\theta \right)$$ is the covariance function matrix (for more information, see Gaussian Process Regression Models).

To estimate the parameters, the software first computes $$\widehat{\beta }\left(\theta ,{\sigma }^2\right)$$, which maximizes the log likelihood function with respect to $$\beta$$ for given $$\theta$$ and $${\sigma }^2$$. It then uses this estimate to compute the $$\beta$$-profiled likelihood:

The estimate of $$\beta$$ for given $$\theta$$, and $${\sigma }^2$$ is

Then, the $$\beta$$-profiled log likelihood is given by

The software then maximizes the $$\beta$$-profiled log-likelihood over $$\theta$$, and $${\sigma }^2$$ to find their estimates.

Prediction

Making probabilistic predictions from a GPR model with known parameters requires the density $$P\left(y_{new}|y,X,x_{new}\right)$$. Using the definition of conditional probabilities, one can write:

To find the joint density in the numerator, it is necessary to introduce the latent variables $$f_{new}$$ and $$f$$ corresponding to $$y_{new}$$, and $$y$$, respectively. Then, it is possible to use the joint distribution for $$y_{new}$$, $$y$$, $$f_{new}$$, and $$f$$ to compute $$P\left(y_{new},y|X,x_{new}\right)$$:

Gaussian process models assume that each response $$y_i$$ only depends on the corresponding latent variable $$f_i$$ and the feature vector $$x_i$$. Writing $$P\left(y_{new},y|f_{new},f,X,x_{new}\right)$$ as a product of conditional densities and based on this assumption produces:

After integrating with respect to $$y_{new}$$, the result only depends on $$f$$ and $$X$$:

Hence,

Again using the definition of conditional probabilities,

it is possible to write $$P\left(y_{new},y|X,x_{new}\right)$$ as follows:

Using the facts that

and

one can rewrite $$P\left(y_{new},y|X,x_{new}\right)$$ as follows:

It is also possible to show that

Hence, the required density $$P\left(y_{new}|y,X,x_{new}\right)$$ is:

It can be shown that

After the integration and required algebra, the density of the new response $$y_{new}$$ at a new point $$x_{new}$$, given $$y$$, $$X$$ is found as

where

and

The expected value of prediction $$y_{new}$$ at a new point $$x_{new}$$ given $$y$$, $$X$$, and parameters $$\beta$$, $$\theta$$, and $${\sigma }^2$$ is

where

Computational Complexity of Exact Parameter Estimation and Prediction

Training a GPR model with the exact method (when FitMethod is 'Exact') requires the inversion of an n-by-n kernel matrix $$K\left(X,X\right)$$. The memory requirement for this step scales as O(n^2) since $$K\left(X,X\right)$$ must be stored in memory. One evaluation of $$\log P\left(y|X\right)$$ scales as O(n^3). Therefore, the computational complexity is O(k*n^3), where k is the number of function evaluations needed for maximization and n is the number of observations.

Making predictions on new data involves the computation of $$\widehat{\alpha }$$. If prediction intervals are desired, this step could also involve the computation and storage of the Cholesky factor of $$\left(K\left(X,X\right)+{\sigma }^2{I}_n\right)$$ for later use. The computational complexity of this step using the direct computation of $$\widehat{\alpha }$$ is O(n^3) and the memory requirement is O(n^2).

Hence, for large n, estimation of parameters or computing predictions can be very expensive. The approximation methods usually involve rearranging the computation so as to avoid the inversion of an n-by-n matrix. For the available approximation methods, please see the related links at the bottom of the page.