[dead]
Embarrassingly simple self-distillation improves code generation
101–110 of 227 posts
Re: Embarrassingly simple self-distillation improves code generation
#102Really fascinating how this works; it's basically context-aware decoding. From the paper: > Code interleaves fork positions, where several continuations are genuinely plausible and may correspond to different solution approaches, with lock positions, where syntax and semantics leave little ambiguity but a low-probability distractor tail still remains… The best global decoding setting is therefore necessarily a compro…
Another example of the mindf@#$ these systems are: I was doing some fine tuning to a small model, take data fields and make a sentence out of it. I was running into mode collapse (basically when the AI simplifies too much and always output the same thing). I got unstuck by randomizing the field order for each row?!? At training, and now I'm thinking I should do the same at inference time...
apparently you can straight up duplicate/add/rearrange layers without changing any of the weights and get better results as well - https://dnhkng.github.io/posts/rys/
Re: Embarrassingly simple self-distillation improves code generation
#103There's an obvious baseline which seems missing
If you sample from the base model with T=1.6, top_k=20, top_p=0.8, i.e, the decode settings used for the distillation's ground truth, does it match the SSD'd model + some decoding? Performance wise.
Their sweep is missing this. And only covers "standard" decoding settings.
Re: Embarrassingly simple self-distillation improves code generation
#104Fascinating... This feels eerily similar to sleep consolidation or synaptic pruning
I don't see much similarity? Unless you're looking at self-distillation in general and not just this use of it.
How not?
I think the analogy is actually pretty specific to this paper, not just self-distillation in general.
During sleep your brain replays experiences but noisy and distorted. The replays are often incoherent as narratives (dreams are weird). But the consolidation still works because the value isn't in the narrative coherence, it's in the activation patterns at each moment. Important pathways get strengthened, weak ones get pruned. Section 4.4 of this paper is what makes the connection click. They cranked training temperature to 2.0 with no truncation. 62% of the sampled outputs had no extractable code. Coherent Python that devolves into multilingual gibberish halfway through. The model still improved (+5.7pp pass@1).
This makes no sense if you think the model is learning from good code examples. But it makes a lot of sense if you think of it as the model replaying its own knowledge back to itself in a noisy/distorted form, and the replay process strengthening what matters (sharp distributions at "lock" positions where one token is correct, broad distributions at "fork" positions where multiple approaches work) while pruning what doesn't (distractor tails). The model doesn't learn anything new. It just wakes up performing better because what it already knew got cleaned up.
How is this comment not at number 1??
Re: Embarrassingly simple self-distillation improves code generation
#105Really fascinating how this works; it's basically context-aware decoding. From the paper: > Code interleaves fork positions, where several continuations are genuinely plausible and may correspond to different solution approaches, with lock positions, where syntax and semantics leave little ambiguity but a low-probability distractor tail still remains… The best global decoding setting is therefore necessarily a compro…
Could we not get the same with EAFT? Maybe that’s what it’s doing but definitely not the first to think “let’s lock in high probability solutions”
In nemotron the high perplexity solutions are selected for RL, in VLM training a few people are looking at the entropy distributions of the training set, etc
Re: Embarrassingly simple self-distillation improves code generation
#106"SSD improves Qwen3-30B-Instruct from 42.4% to 55.3% pass@1 on LiveCodeBench v6"
I know virtually nothing about this area but my naive take is that something that means it still only passes tests around half the time doesn't seem like a particularly big jump forwards.
What am I missing?
Re: Embarrassingly simple self-distillation improves code generation
#107So the chances of Singularity went up.
Or down if this research leads to a local minima.
Re: Embarrassingly simple self-distillation improves code generation
#108Earlier quoted context omitted.
> I love that we're still learning the emergent properties of LLMs! TBH, this is (very much my opinion btw) the least surprising thing. LLMs (and especially their emergent properties) are still black boxes. Humans have been studying the human brain for millenia, and we are barely better at predicting how humans work (or for eg to what extent free will is a thing). Hell, emergent properties of traffic was not understo…
Studies of LLMs belong in their own field of science, just like psychology is not being studied in the physics department.
Interestingly enough, for a while physics used to be studied by philosophers (and used to be put in the natural philosophy basket, together with biology and most other hard sciences).
Re: Embarrassingly simple self-distillation improves code generation
#109Earlier quoted context omitted.
Another example of the mindf@#$ these systems are: I was doing some fine tuning to a small model, take data fields and make a sentence out of it. I was running into mode collapse (basically when the AI simplifies too much and always output the same thing). I got unstuck by randomizing the field order for each row?!? At training, and now I'm thinking I should do the same at inference time...
apparently you can straight up duplicate/add/rearrange layers without changing any of the weights and get better results as well - https://dnhkng.github.io/posts/rys/
This is crazy, thank you for the link!
Re: Embarrassingly simple self-distillation improves code generation
#110Earlier quoted context omitted.
Studies of LLMs belong in their own field of science, just like psychology is not being studied in the physics department.
The intersection of physics isnt psychology it is philosophy, and the same is true (at present) with LLM's Much as Diogenes mocked Platos definition of a man with a plucked chicken, LLM's revealed what "real" ai would require: contigous learning. That isnt to diminish the power of LLM's (the are useful) but that limitation is a fairly hard one to over come if true AGI is your goal.
Is it because we haven't invented something better than backpropagation yet?
From what I understand, a living neural network learns several orders of magnitude more efficiently than an artificial one.
I'm not sure where that difference comes from. But my brain probably isn't doing back propagation, it's probably doing something very different.