Gemini Distillation Service

bizzletk wrote: post #5

moralestapia wrote: post #4

Fine-tuning and distillation used to mean two different things. Now not so much anymore, I see.

Can you explain the difference?

Fine-tuning is done against a dataset. Distillation is done against a model.

Re: Gemini Distillation Service

#7

Post by porridgeraisin » Tue, Jul 28, 2026, 5:57 AM UTC

bizzletk wrote: post #5

moralestapia wrote: post #4

Fine-tuning and distillation used to mean two different things. Now not so much anymore, I see.

Can you explain the difference?

Distillation originally meant matching the distribution of the student model to the teacher model using something like a KL divergence.

When you instead fine-tune the student on the samples from the teacher, which is what people mean by distillation today, you are in effect doing a monte-carlo version of the same thing. While in theory this is higher variance, given modern setups where the student and teacher are both large and are RLd heavily (leading to a sharp teacher distribution), and given that you typically use lots and lots of samples, it ends up OK.

Re: Gemini Distillation Service

#8

Post by sapienskid » Wed, Jul 29, 2026, 2:46 AM UTC

moralestapia wrote: post #4

Fine-tuning and distillation used to mean two different things. Now not so much anymore, I see.

How to get access?

Re: Gemini Distillation Service

#9

Post by rs38 » Wed, Jul 29, 2026, 6:31 AM UTC

404?

Re: Gemini Distillation Service

#10

Post by verdverm » Wed, Jul 29, 2026, 5:59 PM UTC

rs38 wrote: post #9

404?

just heard on Risky Business pod that they apparently have taken it down, I distinctly remember reading it two days ago, came here to find the link, found your 404?