Embarrassingly simple self-distillation improves code generation • Hacker News Board

Re: Embarrassingly simple self-distillation improves code generation

#161

Post by astrange » Sat, Apr 04, 2026, 9:35 PM UTC

danielmarkbruce wrote: post #138

batshit_beaver wrote: post #121

Earlier quoted context omitted.

> We "literally" couldn't design programs to do the interesting things LLMs can do. That's a bit of an overstatement. The entire field of ML is aimed at problems where deterministic code would work just fine, but the amount of cases it would need to cover is too large to be practical (note, this has nothing to do with the impossibility of its design) AND there's a sufficient corpus of data that allows plausible enoug…

Saying ML is a field where deterministic code would work just fine conveniently leaves out the difficult part - writing the actual code.... Which we haven't been able to do for most of the tasks at hand. What you are saying is fantasy nonsense.

They did not leave it out.

> but the amount of cases it would need to cover is too large to be practical (note, this has nothing to do with the impossibility of its design)

Re: Embarrassingly simple self-distillation improves code generation

#162

Post by astrange » Sat, Apr 04, 2026, 9:37 PM UTC

andai wrote: post #110

zer00eyz wrote: post #76

Earlier quoted context omitted.

The intersection of physics isnt psychology it is philosophy, and the same is true (at present) with LLM's Much as Diogenes mocked Platos definition of a man with a plucked chicken, LLM's revealed what "real" ai would require: contigous learning. That isnt to diminish the power of LLM's (the are useful) but that limitation is a fairly hard one to over come if true AGI is your goal.

Is it because we haven't invented something better than backpropagation yet? From what I understand, a living neural network learns several orders of magnitude more efficiently than an artificial one. I'm not sure where that difference comes from. But my brain probably isn't doing back propagation, it's probably doing something very different.

Your brain is doing several different things, because there are different parts of your brain.

(eg different kinds of learning for long-term memory, short-term memory, languages, faces and reflexes.)

Re: Embarrassingly simple self-distillation improves code generation

#163

Post by astrange » Sat, Apr 04, 2026, 9:38 PM UTC

devmor wrote: post #143

sgbeal wrote: post #98

Earlier quoted context omitted.

> Give coding agents access to intellisense and syntax highlighting. i once asked an LLM if it could ingest code from an interactive session more easily if it were in appropriately-typed markdown fences and it said absolutely yes, and that the syntax highlighting fed to it that way helps it immensely. i was downright shocked that syntax highlighting was anything more than noise for them.

Why would this be surprising? That’s exactly how much of the code they were trained on is presented in PRs, Forums, etc.

Is that true? That depends on how their web scraping works, like whether it runs client-side highlighting, strips out HTML tags, etc.

Re: Embarrassingly simple self-distillation improves code generation

#164

Post by hnretards » Sat, Apr 04, 2026, 9:38 PM UTC

I've been doing something even better than this for years using only Mistral 7b.

My local running Mistral 7b is a 100x better at modern JavaScript than any model on the market, mainly just from RAG on my own code samples.

That's basically what they are describing with "post-training", the TLDR is that code especially of a certain style is vastly simpler than written language.

You really don't need a huge model or data centers etc. you just need a small but good model like Mistral 7b and literally a few good samples.

But you guys keep doing you lol. A bunch of non-devs trying to solve code is pretty funny to watch.

Re: Embarrassingly simple self-distillation improves code generation

#165

Post by astrange » Sat, Apr 04, 2026, 9:39 PM UTC

p_stuart82 wrote: post #73

khalic wrote: post #42

Earlier quoted context omitted.

Another example of the mindf@#$ these systems are: I was doing some fine tuning to a small model, take data fields and make a sentence out of it. I was running into mode collapse (basically when the AI simplifies too much and always output the same thing). I got unstuck by randomizing the field order for each row?!? At training, and now I'm thinking I should do the same at inference time...

the irony of modern software engineering: we spent decades perfecting deterministic algorithms, and now we're basically just shaking a black box and hoping the magic rocks align.

This is an AI bot btw. (sarcasm, metaphor that doesn't make sense)

Re: Embarrassingly simple self-distillation improves code generation

#166

Post by drooby » Sat, Apr 04, 2026, 9:46 PM UTC

ACCount37 wrote: post #141

oliver236 wrote: post #104

Earlier quoted context omitted.

How not? I think the analogy is actually pretty specific to this paper, not just self-distillation in general. During sleep your brain replays experiences but noisy and distorted. The replays are often incoherent as narratives (dreams are weird). But the consolidation still works because the value isn't in the narrative coherence, it's in the activation patterns at each moment. Important pathways get strengthened, we…

This is a property of self-distillation. Self-distillation shifts the behavior of the model towards that of the model + steering. As such, you don't strictly "need" the tokens to be in-domain for it to work. The logits are a vessel for transferring the steering into the model's internals. The tokens can be gibberish. What transfers isn't whether they're gibberish or not, but how the flavor of model predictions, if gi…

I think we’re agreeing. The point of the sleep parallel is exactly that the content doesn’t matter, and it’s the filtering process that does the work. Brains replay noisy, sometimes incoherent patterns during sleep and the value is in how that replay reshapes connection weights, not in whether the replay is accurate. That’s the same principle you’re describing with the steering signal

I.e sleep replays don’t need to replay Tuesday’s meeting accurately. They just need to activate the relevant pathways so that the strong ones fire and the weak ones don’t. The pattern of what fires versus what doesn’t is the signal. The “content” of the dream is basically irrelevant.

Re: Embarrassingly simple self-distillation improves code generation

#167

Post by olejorgenb » Sat, Apr 04, 2026, 9:47 PM UTC

vova_hn2 wrote: post #89

bensyverson wrote: post #16

Really fascinating how this works; it's basically context-aware decoding. From the paper: > Code interleaves fork positions, where several continuations are genuinely plausible and may correspond to different solution approaches, with lock positions, where syntax and semantics leave little ambiguity but a low-probability distractor tail still remains… The best global decoding setting is therefore necessarily a compro…

I've always thought that it is kinda weird that we spend exactly the same amount of compute to calculate both "fork" tokens and "lock" tokens. I think that with grammar-aware sampling / constrained decoding [0][1] it is possible to sometimes skip calling the model altogether if only one token is allowed by grammar and just insert it, but I don't think that any of the current, widely used combinations of models/harnes…

> I wonder if there is a more general solution that can make models spend more compute on making important choices, while making generation of the "obvious" tokens cheaper and faster.

I think speculative decoding count as a (perhaps crude) way implementing this?

Re: Embarrassingly simple self-distillation improves code generation

#168

Post by khalic » Sat, Apr 04, 2026, 9:49 PM UTC

astrange wrote: post #165

p_stuart82 wrote: post #73

Earlier quoted context omitted.

the irony of modern software engineering: we spent decades perfecting deterministic algorithms, and now we're basically just shaking a black box and hoping the magic rocks align.

This is an AI bot btw. (sarcasm, metaphor that doesn't make sense)

Me or the new account?

Re: Embarrassingly simple self-distillation improves code generation

#169

Post by hooloovoo_zoo » Sat, Apr 04, 2026, 9:53 PM UTC

One sentence summary: We fine-tuned a general-purpose model to produce valid benchmark code results and it got better at producing benchmark code results; we didn't bother to evaluate it on anything the model used to be good at.

Re: Embarrassingly simple self-distillation improves code generation

#170

Post by andy_xor_andrew » Sat, Apr 04, 2026, 10:04 PM UTC

hooloovoo_zoo wrote: post #169

One sentence summary: We fine-tuned a general-purpose model to produce valid benchmark code results and it got better at producing benchmark code results; we didn't bother to evaluate it on anything the model used to be good at.

Not really? If you read it, there is no validation, no correctness signal, no verification, none of that. They're just passing in benchmark inputs, collecting the outputs (regardless of their quality), training on those outputs, and then sweeping the decode settings (temp, topk) of the resulting model. Their conclusion is that this results in a better model than the original - even when taking into consideration the same temp/topk sweep of the original.

So no, they are not fine-tuning a general purpose model to produce "valid benchmark code results."