Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Embarrassingly simple self-distillation improves code generation

arxiv.org

Post reply on HN
Print view View original

221–227 of 227 posts

Re: Embarrassingly simple self-distillation improves code generation

#221

Post by user_7832 »

bensyverson wrote: post #96
user_7832 wrote: post #92

Earlier quoted context omitted.

Oh I very much agree that it's great to see more research and findings and improvements in this field. I'm just a little puzzled by GP's tone (which suggested that it isn't completely expected to find new things about LLMs, a few years in).

I'm the GP! lol… Not sure how you got that from my tone, but I find these discoveries expected but not routine, and also interesting.

Sorry lol, to me it felt like you were (pleasantly) surprised by this research. IMO I'd hardly be surprised to see breakthroughs in LLM understanding years or even decades from now. I guess I misunderstood your tone.

Re: Embarrassingly simple self-distillation improves code generation

#222

Post by user_7832 »

Invictus0 wrote: post #151
user_7832 wrote: post #95

Earlier quoted context omitted.

I hate to "umm, akshually" but apparently we have been studying the brain for thousands of years. I wasn't talking about purely modern neuroscience (which ironically for our topic of emergence, (often till recently/still in most places) treats the brain as the sum of its parts - be them neurons or neurotransmitters). > The earliest reference to the brain occurs in the Edwin Smith Surgical Papyrus, written in the 17th…

None of that counts as studying the brain. It's like saying rubbing sticks together to make fire counts as studying atomic energy. Those early "researchers" were hopelessly far away from even the most tangential understanding of the workings of the brain.

But fundamentally speaking, they were trying to understand the brain, right? IMO that counts as science/study in my books. They understood parts/basics of intracranial pressure so long back.

And if we say it's not science if it's not correct, well, (modern) physics isn't a science then, right? ;) As we haven't unified relativity with quantum mechanics?

Re: Embarrassingly simple self-distillation improves code generation

#223

Post by yesexactly »

l5870uoo9y wrote: post #23

> Our method, simple self-distillation (SSD), is embarrassingly simple: sample solutions from the base model with specified temperature and truncation, then fine-tune on those raw, unverified samples via standard cross-entropy loss. So you prompt the base model for answer and then rerun the prompt with the answer from the first run?

Yeah basically.

It's annoying as hell how much euphemistic language is used.

They say "embarassingly simple" but they really mean "something everyone already knows"

They have made 0 discoveries

Re: Embarrassingly simple self-distillation improves code generation

#224

Post by HarHarVeryFunny »

bensyverson wrote: post #16

Really fascinating how this works; it's basically context-aware decoding. From the paper: > Code interleaves fork positions, where several continuations are genuinely plausible and may correspond to different solution approaches, with lock positions, where syntax and semantics leave little ambiguity but a low-probability distractor tail still remains… The best global decoding setting is therefore necessarily a compro…

Apparently a key part of this is not just to use the combination of high temperature (to boost fork diversity) and top-k (to truncate unwanted diversity at lock positions) sampling, but rather to use these settings to first generate a fine tuning dataset and then train on that. The fine tuning lets the model adapt it's weights to the new skewed distribution, which sounds a bit like an annealing process.

It does raise some questions:

1) Is this always a win for coding? The top-k truncation is also going to limit "fork" diversity. Maybe there is a better way to reshape the output probability distribution that sharpens the cutoff where it is already sharp (locks), without affecting it so much where it is more gradual (forks)?

2) Wouldn't this also benefit generation for other non-coding domains, which are generally also going to contain both "fork" and "lock" positions?

Re: Embarrassingly simple self-distillation improves code generation

#226

Post by 3rd3 »

ACCount37 wrote: post #30
l5870uoo9y wrote: post #23

> Our method, simple self-distillation (SSD), is embarrassingly simple: sample solutions from the base model with specified temperature and truncation, then fine-tune on those raw, unverified samples via standard cross-entropy loss. So you prompt the base model for answer and then rerun the prompt with the answer from the first run?

No. There's no "answer" really. They use self-distillation to shift the output distribution of the model towards that of the same model, but running with different temperature/truncation settings in sampling. This effectively "folds" the logit tail truncation behavior into the model itself. Not entirely unlike a few "model controlled sampling settings" things I've seen in what it does, but different in execution.

Isn't that "scheduled sampling"? In that case they also shift the input distribution toward that of the model, which possibly is even more crucial than shifting the output distribution?

Re: Embarrassingly simple self-distillation improves code generation

#227

Post by foota »

jameshart wrote: post #189
foota wrote: post #186

Earlier quoted context omitted.

Not really, because the LLM loop doesn't have the ability to get updates from the agent live. It would have to somehow be integrated all the way down the stack.

LLMs can have whatever abilities we build for them. The fact we currently start their context out with a static prompt which we keep feeding in on every iteration of the token prediction loop is a choice. We don’t have to keep doing that if there are other options available.

Late to reply, but.. yes, hence my reply that it would need to be integrated all the way down the stack.

But also, LLMs (or their current implementation) rely heavily on print caching for efficiency, without this costs are much higher. You can do neat tricks with it, but generally you're limited to playing with the end of the context to avoid breaking things.

I think some agents do add small context snippets to the end of the conversation that get used by the agent. You can do things like: conversation messages + context snippets + new message and then once the agent replies make the next turn conversation + new message + reply + ... This breaks the cache only for the latest message (not too bad) and let's you give the model current up to date information. This is how stuff like the "mode" or "what time is it now" are handled I believe.

Post reply on HN