Gaussian process

We have the follow impractical definition:

A Gaussian process is a collection of random variables, any finite number of which have consistent Gaussian distributions.

It is completely defined by its mean function $m(x)$ and covariance function $\Sigma (x,x^{\prime })$. We write:

\begin{equation}
  f(x) \sim \mathcal{G}\left(m(x), \Sigma(x, x')\right)
\end{equation}

A very import special case of Gaussian processes is Gaussian Markov processes

Kernels

We often express the covariance function $\Sigma (x,x^{\prime })$ as a function of a kernel $K(x,x^{\prime })$):

\begin{align*}
\displaystyle
 \Sigma(x,x') = K(x, x') + \;\mathbb{I}
\end{align*}

• Squared exponential kernel
• Ornstein-Uhlenbeck kernel
• Wiener kernel
• Periodic kernel

We can combine kernels. The multiplication of two covariance functions is a valid covariance functions, the addition of two correlation functions is a correlation function.

References

• The Kernel Cookbook (David Duvenaud’s website)
• Structure Discovery in Nonparametric Regression Through Compositional Search (ArXiV)
• Gaussian Processes for Machine Learning, Chapter 4: Covariance Functions (pdf)
• Kernel Design (slides of Nicolas Durrande’s talk)

Sample

• How to Sample from a Gaussian Process predictive distribution
• Practical implementation of Gaussian Process Regression (Gregory Undersen’s blog)
• Gaussian Processes for Machine Learning, Chapter 2: Regression (pdf)

Learn

• Visual exploration of gaussian processes is a very good introduction
• Gaussian Process, not quite for dummies is excellent, starts from gaussians to gaussian in high dimensions to get to gaussian process. One of the best things I’ve read on the topic.
• (rasmussen2003) is a useful reference, but more advanced

Use

Here is a list of libraries implementating different flavors of Gaussian processes. The catch is that gaussian processes can quick get computation-intensive.

• BayesNewton is written in JAX
• GPytorch is built atop the Pytorch autodiff framework
• PyMC3 also has an implementation of gaussian processes