Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Embarrassingly simple self-distillation improves code generation

arxiv.org

Post reply on HN
Print view View original

41–50 of 227 posts

Re: Embarrassingly simple self-distillation improves code generation

#41

Post by BarryMilo »

spiderfarmer wrote: post #34
wg0 wrote: post #20

After TurboQuant and Gemma 4, came across the following video[0] running Gemma on local machine at 50 token/second. That already looks like Sonnet 3x and 4 level capabilities to me where the model in question (Gemma 4) set ups whole python project with a UI and installs python libraries using uv etc. Add this Simple Self Distillation to the picture and by 2028 I see cheaper coding model providers with much more gener…

I always wonder how much smaller and faster models could be if they were only trained on the latest versions of the languages I use, so for me that is PHP, SQL, HTML, JS, CSS, Dutch, English, plus tool use for my OS of choice (MacOS). Right now it feels like hammering a house onto a nail instead of the other way around.

I seem to remember that's one of the first things they tried, but the general models tended to win out. Turns out there's more to learn from all code/discussions than from just JS.

Re: Embarrassingly simple self-distillation improves code generation

#42

Post by khalic »

bensyverson wrote: post #16

Really fascinating how this works; it's basically context-aware decoding. From the paper: > Code interleaves fork positions, where several continuations are genuinely plausible and may correspond to different solution approaches, with lock positions, where syntax and semantics leave little ambiguity but a low-probability distractor tail still remains… The best global decoding setting is therefore necessarily a compro…

Another example of the mindf@#$ these systems are: I was doing some fine tuning to a small model, take data fields and make a sentence out of it. I was running into mode collapse (basically when the AI simplifies too much and always output the same thing).

I got unstuck by randomizing the field order for each row?!? At training, and now I'm thinking I should do the same at inference time...

Re: Embarrassingly simple self-distillation improves code generation

#43

Post by DavidPiper »

bensyverson wrote: post #16

Really fascinating how this works; it's basically context-aware decoding. From the paper: > Code interleaves fork positions, where several continuations are genuinely plausible and may correspond to different solution approaches, with lock positions, where syntax and semantics leave little ambiguity but a low-probability distractor tail still remains… The best global decoding setting is therefore necessarily a compro…

Sounds just like John Cleese's "Open Mode" and "Closed Mode" - https://www.youtube.com/watch?v=Pb5oIIPO62g

Re: Embarrassingly simple self-distillation improves code generation

#44

Post by [deleted] »

Aurornis wrote: post #19
politelemon wrote: post #10

It's cringe worthy to see that the original paper itself is editorialised. Title should be: Simple Self-Distillation Improves Code Generation

The phrase embarrassingly parallel has a history in computer science. Many computer science paper titles allude to past titles in other CS papers. Calling it “cringe worthy” is unnecessarily mean. There is context and history you don’t understand.

[deleted]

Re: Embarrassingly simple self-distillation improves code generation

#46

Post by nareyko »

spiderfarmer wrote: post #34
wg0 wrote: post #20

After TurboQuant and Gemma 4, came across the following video[0] running Gemma on local machine at 50 token/second. That already looks like Sonnet 3x and 4 level capabilities to me where the model in question (Gemma 4) set ups whole python project with a UI and installs python libraries using uv etc. Add this Simple Self Distillation to the picture and by 2028 I see cheaper coding model providers with much more gener…

I always wonder how much smaller and faster models could be if they were only trained on the latest versions of the languages I use, so for me that is PHP, SQL, HTML, JS, CSS, Dutch, English, plus tool use for my OS of choice (MacOS). Right now it feels like hammering a house onto a nail instead of the other way around.

[dead]

Re: Embarrassingly simple self-distillation improves code generation

#47

Post by an0malous »

I’d like to understand AI research better and I recall some posts a while back where someone collected all the key papers that one should read, but I don’t remember enough to be able to find it. Does anyone know what I’m talking about and could link me to that post?

Re: Embarrassingly simple self-distillation improves code generation

#48

Post by Someone1234 »

spiderfarmer wrote: post #34
wg0 wrote: post #20

After TurboQuant and Gemma 4, came across the following video[0] running Gemma on local machine at 50 token/second. That already looks like Sonnet 3x and 4 level capabilities to me where the model in question (Gemma 4) set ups whole python project with a UI and installs python libraries using uv etc. Add this Simple Self Distillation to the picture and by 2028 I see cheaper coding model providers with much more gener…

I always wonder how much smaller and faster models could be if they were only trained on the latest versions of the languages I use, so for me that is PHP, SQL, HTML, JS, CSS, Dutch, English, plus tool use for my OS of choice (MacOS). Right now it feels like hammering a house onto a nail instead of the other way around.

Wouldn't that mean they're bad at migration tasks? I feel like for most languages, going from [old] to [current] is a fairly to very common usage scenario.

Re: Embarrassingly simple self-distillation improves code generation

#49

Post by TacticalCoder »

bensyverson wrote: post #16

Really fascinating how this works; it's basically context-aware decoding. From the paper: > Code interleaves fork positions, where several continuations are genuinely plausible and may correspond to different solution approaches, with lock positions, where syntax and semantics leave little ambiguity but a low-probability distractor tail still remains… The best global decoding setting is therefore necessarily a compro…

> What this paper shows is that their simple technique (SSD)

"Simple Self-Distillation". We had an acronym for Solid-State Drive. Don't know about that technique but the naming sure sound.. Simple?

Re: Embarrassingly simple self-distillation improves code generation

#50

Post by bensyverson »

user_7832 wrote: post #40
bensyverson wrote: post #16

Really fascinating how this works; it's basically context-aware decoding. From the paper: > Code interleaves fork positions, where several continuations are genuinely plausible and may correspond to different solution approaches, with lock positions, where syntax and semantics leave little ambiguity but a low-probability distractor tail still remains… The best global decoding setting is therefore necessarily a compro…

> I love that we're still learning the emergent properties of LLMs! TBH, this is (very much my opinion btw) the least surprising thing. LLMs (and especially their emergent properties) are still black boxes. Humans have been studying the human brain for millenia, and we are barely better at predicting how humans work (or for eg to what extent free will is a thing). Hell, emergent properties of traffic was not understo…

Learning about the emergent properties of these black boxes is not surprising, but it's also not daily. I think every new insight is worth celebrating.
Post reply on HN