Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Embarrassingly simple self-distillation improves code generation

arxiv.org

Post reply on HN
Print view View original

171–180 of 227 posts

Re: Embarrassingly simple self-distillation improves code generation

#171

Post by andy_xor_andrew »

zyklu5 wrote: post #139

Their explanation for why their idea (SSD) might work - precision-exploration conflict hypothesis - is something adaptive decoding also tries to solve. https://ai.meta.com/research/publications/adaptive-decoding-...

I've been wondering about adaptive decoding! It seems obvious to me that at some points during decoding (reasoning, "creative thinking") you would want a higher temperature, while at other points (emitting syntactically correct code, following a plan that was already established) you would want lower temperature.

Re: Embarrassingly simple self-distillation improves code generation

#172

Post by Lerc »

This is the natural conclusion of what was really claimed about model collapse, and indeed natural evolution. Making an imperfect copy while invoking a selection mechanism is evolution.

Some of the claims about models training on their own data, in their enthusiasm to frame it as a failure, went further to suggest that it magnified biases. I had my doubts about their conclusions. If it were true, it would be a much greater breakthrough because the ability to magnify a property represents a way to measure a weak version that property. The ability to do that would mean they would have found a way to provide a training signal to avoid bias. It would be great if that's what they did but I suspect there would have been more news about it.

Perhaps this paper will put to rest the notion that AI output is useless as training data. It has only ever been the case that it was useless as an indiscriminate source of data.

Re: Embarrassingly simple self-distillation improves code generation

#173

Post by jditu »

uduni wrote: post #120

It's crazy how much better you can make LLM output just by asking "is this the most elegant solution?" In a loop (Not fine tuning, but interesting none the less. If a model can so easily find a more elegant solution, why didn't it pick that in the first place?)

[dead]

Re: Embarrassingly simple self-distillation improves code generation

#174

Post by hooloovoo_zoo »

andy_xor_andrew wrote: post #170
hooloovoo_zoo wrote: post #169

One sentence summary: We fine-tuned a general-purpose model to produce valid benchmark code results and it got better at producing benchmark code results; we didn't bother to evaluate it on anything the model used to be good at.

Not really? If you read it, there is no validation, no correctness signal, no verification, none of that. They're just passing in benchmark inputs, collecting the outputs (regardless of their quality), training on those outputs, and then sweeping the decode settings (temp, topk) of the resulting model. Their conclusion is that this results in a better model than the original - even when taking into consideration the…

They are training the model to 1. Produce code (as opposed to answer a question, write a poem, etc.) 2. Produce long enough output to be a valid solution. So they are doing exactly what I said. Cheers.

Re: Embarrassingly simple self-distillation improves code generation

#175

Post by mememememememo »

hooloovoo_zoo wrote: post #174
andy_xor_andrew wrote: post #170

Earlier quoted context omitted.

Not really? If you read it, there is no validation, no correctness signal, no verification, none of that. They're just passing in benchmark inputs, collecting the outputs (regardless of their quality), training on those outputs, and then sweeping the decode settings (temp, topk) of the resulting model. Their conclusion is that this results in a better model than the original - even when taking into consideration the…

They are training the model to 1. Produce code (as opposed to answer a question, write a poem, etc.) 2. Produce long enough output to be a valid solution. So they are doing exactly what I said. Cheers.

In layman, they are putting wet tyres on when it is raining and saying the car performs better over the next lap?

Re: Embarrassingly simple self-distillation improves code generation

#176

Post by noman-land »

uduni wrote: post #120

It's crazy how much better you can make LLM output just by asking "is this the most elegant solution?" In a loop (Not fine tuning, but interesting none the less. If a model can so easily find a more elegant solution, why didn't it pick that in the first place?)

The elegant solution rarely happens on the first try. Many times you need to first arrive at a solution, and then keep iterating on it until it's elegant. Akin to "sorry I didn't have time to write a shorter letter".

Re: Embarrassingly simple self-distillation improves code generation

#178

Post by [deleted] »

devmor wrote: post #143
sgbeal wrote: post #98

Earlier quoted context omitted.

> Give coding agents access to intellisense and syntax highlighting. i once asked an LLM if it could ingest code from an interactive session more easily if it were in appropriately-typed markdown fences and it said absolutely yes, and that the syntax highlighting fed to it that way helps it immensely. i was downright shocked that syntax highlighting was anything more than noise for them.

Why would this be surprising? That’s exactly how much of the code they were trained on is presented in PRs, Forums, etc.

[deleted]

Re: Embarrassingly simple self-distillation improves code generation

#179

Post by sdwr »

bensyverson wrote: post #16

Really fascinating how this works; it's basically context-aware decoding. From the paper: > Code interleaves fork positions, where several continuations are genuinely plausible and may correspond to different solution approaches, with lock positions, where syntax and semantics leave little ambiguity but a low-probability distractor tail still remains… The best global decoding setting is therefore necessarily a compro…

What's cool is that they aren't adjusting the temperature of the model live, or predicting/labeling any of the fork/lock points.
Post reply on HN