[flagged]
[deleted]
Embarrassingly simple self-distillation improves code generation
11–20 of 227 posts
Re: Embarrassingly simple self-distillation improves code generation
#12Shouldn't a scientific paper be using metric units (like 30T) rather than 30B.
There are two distinct billions. https://en.wikipedia.org/wiki/Billion
Re: Embarrassingly simple self-distillation improves code generation
#13[flagged]
So
"Made in China, designed by Apple in California"
should be:
"Made in China, designed by Chinese people in California"?
Re: Embarrassingly simple self-distillation improves code generation
#14Skimmed this but don't have an intuitive understanding of why this works and how temperature and truncation factor in.
Re: Embarrassingly simple self-distillation improves code generation
#15[flagged]
I used to have the opposite rule in my signal processing field : the more Chinese names, the less innovation was there.
They seemed like they had to be churning out papers and any little adaptation to existing research triggered a new publication.
But it may have changed now.
Re: Embarrassingly simple self-distillation improves code generation
#16Really fascinating how this works; it's basically context-aware decoding. From the paper:
> Code interleaves fork positions, where several continuations are genuinely plausible and may correspond to different solution approaches, with lock positions, where syntax and semantics leave little ambiguity but a low-probability distractor tail still remains… The best global decoding setting is therefore necessarily a compromise; we call this tension the precision-exploration conflict.
In other words, just like us, the model needs to shift from "exploration" in "fork" mode (divergent thinking to produce a creative solution) to "precision" in "lock" mode (producing syntactically correct code).
What this paper shows is that their simple technique (SSD) can improve the ranking of optimal tokens in both lock and fork positions, meaning the model is more likely to explore when it should be exploring, and more likely to be precise when it needs to be.
I love that we're still learning the emergent properties of LLMs!
Re: Embarrassingly simple self-distillation improves code generation
#17It's cringe worthy to see that the original paper itself is editorialised. Title should be: Simple Self-Distillation Improves Code Generation
"Embarrassingly" has a history as a technically meaningful word roughly equivalent to "maximally", see "Embarrassingly parallel"
Re: Embarrassingly simple self-distillation improves code generation
#18[flagged]
I definitely pay more attention to papers affiliated with Chinese companies; the economics seem to be more conducive to doing good academic work and publishing it. I would say the same for companies like Apple (where TFA came from).
But to filter based on author's names sounds pretty darn racist.
Re: Embarrassingly simple self-distillation improves code generation
#19It's cringe worthy to see that the original paper itself is editorialised. Title should be: Simple Self-Distillation Improves Code Generation
The phrase embarrassingly parallel has a history in computer science.
Many computer science paper titles allude to past titles in other CS papers.
Calling it “cringe worthy” is unnecessarily mean. There is context and history you don’t understand.
Re: Embarrassingly simple self-distillation improves code generation
#20After TurboQuant and Gemma 4, came across the following video[0] running Gemma on local machine at 50 token/second.
That already looks like Sonnet 3x and 4 level capabilities to me where the model in question (Gemma 4) set ups whole python project with a UI and installs python libraries using uv etc.
Add this Simple Self Distillation to the picture and by 2028 I see cheaper coding model providers with much more generous usage limits in the future and power users would be mostly running their own models anyway.
Anyone using these models as "non-deterministic transpilers" from natural language to code (experienced engineers who can write code themselves) would probably not be paying to any AI providers.