Gaussian process
We have the follow impractical definition:
A Gaussian process is a collection of random variables, any finite number of which have consistent Gaussian distributions.
It is completely defined by its mean function \(m(x)\) and covariance function \(\Sigma(x, x')\). We write:
\begin{equation} f(x) \sim \mathcal{G}\left(m(x), \Sigma(x, x')\right) \end{equation}A very import special case of Gaussian processes is Gaussian Markov processes
Kernels
We often express the covariance function \(\Sigma(x, x')\) as a function of a kernel \(K(x,x')\)):
\begin{align*} \displaystyle \Sigma(x,x') = K(x, x') + \sigma^{2}_{y}\;\mathbb{I} \end{align*}We can combine kernels. The multiplication of two covariance functions is a valid covariance functions, the addition of two correlation functions is a correlation function.
References
- The Kernel Cookbook (David Duvenaud's website)
- Structure Discovery in Nonparametric Regression Through Compositional Search (ArXiV)
- Gaussian Processes for Machine Learning, Chapter 4: Covariance Functions (pdf)
- Kernel Design (slides of Nicolas Durrande's talk)
Sample
- How to Sample from a Gaussian Process predictive distribution
- Practical implementation of Gaussian Process Regression (Gregory Undersen's blog)
- Gaussian Processes for Machine Learning, Chapter 2: Regression (pdf)
Learn
- Visual exploration of gaussian processes is a very good introduction
- Gaussian Process, not quite for dummies is excellent, starts from gaussians to gaussian in high dimensions to get to gaussian process. One of the best things I've read on the topic.
- cite:rasmussen2003 is a useful reference, but more advanced
Use
Here is a list of libraries implementating different flavors of Gaussian processes. The catch is that gaussian processes can quick get computation-intensive.
- BayesNewton is written in JAX
- GPytorch is built atop the Pytorch autodiff framework
- PyMC3 also has an implementation of gaussian processes