A visual introduction to kernel functions

Post by saeranv » Tue, Jun 09, 2026, 5:09 PM UTC

alok-g wrote: post #17

saeranv wrote: post #15

Earlier quoted context omitted.

> Why is the 'true process' changing here? I understand our best guess or model is changing with new observations, but the true process should not be changing. If it actually is, then the formulation should be changed to isolate the parameters that is feeding back to it. He's not saying the true process is changing, just the functions that are being sampled from the GP. The true process refers to the true, underlying…

Thanks. I read several times, and along with another response, I think I have a better understanding now, though still not having a complete grasp. >> So sampling one point gives us the gold amount for cheese amount 1, 2, and 3. This is the 'function', and ... I get this part, so each point in this N-dimensional space yields a function f of the index, and this is the function. >> Yes, the function changes shape as yo…

I said before that the function shape changes as you're updating the parameters that govern the function but that's actually very misleading, (sorry), since the kernel parameters are only indirectly governing the function. What the parameters directly govern is the joint probability distribution P(f(x1), f(x2), ..., f(xn)). So the function f is implicitly defined by how likely the entire sequence of f values are.

So how does it change shape? Well this part is actually something I don't fully grasp myself yet. But I can sketch a crude bayesian interpretation, which is how I think of it. Not completely correct but works as a placeholder until I fully work out the math of updating the parameters.

Basically, from a bayesian perspective we can condition the joint distribution of function outputs as a likelihood conditioned on the kernel parameters theta: p(f(x1), f(x2), ... | theta).

Then we can derive the posterior distribution over theta p(theta | f(x1), f(x2), ...) like so:

p(theta | f(x1), f(x2), ...) ≈ p(f(x1), f(x2), ... | theta) p(theta).

So we fit the theta parameters based on how well it fits the observed data we feed our bayesian model.

FWIW, I recommend chapter 14 of Richard McElreath's Statistical Rethinking for a better introduction of GPs. This article kind of glosses over a lot of the intuition and introductory concepts that you need to really grok it.

Post by alok-g » Wed, Jun 10, 2026, 2:38 AM UTC

magicalhippo wrote: post #20

magicalhippo wrote: post #18

Earlier quoted context omitted.

> k(x, x') = Cov[f(x), f(x')] = Cov[f(r_1 + x * r_2), f(r_1 + x' * r_2)]. As I understand it, it would instead be k(x, x') = Cov[f(x), f(x')] = Cov[r_1 + x * r_2, r_1 + x' * r_2] I admit I haven't run through the full math. Given the definition of covariance I see how you get the x * x' term, but you're right in that it's not immediately obvious the other parts cancel fully.

So working the math a bit, it seems clear the author implicitly assumes the random variables follows a standard normal distribution, so zero mean (E[r] = 0) and unit variance (Var(r) = 1). In that case, you end up with a lot of E[...] = 0 and Var(...) = 1 terms and are left with the x*x' cross term.

And then in the general case, the answer I guess would be a additive superposition of multiple functions including x.x' ... Hence, x.x' serves for purposes of explanation that the original author is aiming for.

Re: A visual introduction to kernel functions

Re: A visual introduction to kernel functions