Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Embarrassingly simple self-distillation improves code generation

arxiv.org

Post reply on HN
Print view View original

211–220 of 227 posts

Re: Embarrassingly simple self-distillation improves code generation

#211

Post by jeremyjh »

da_chicken wrote: post #193
jeremyjh wrote: post #115

Earlier quoted context omitted.

We've been studying brains a lot longer. LLMs are grown, not built. The part that is designed are the low-level architecture - but what it builds from that is incomprehensible and unplanned.

It's not that much longer, really. LLMs draw origins from, both n-gram language models (ca. 1990s) and neural networks and deep learning (ca. 2000). So we've only had really good ones maybe 6-8 years or so, but the roots of the study go back 30 years at least. Psychiatry, psychology, and neurology on the other hand, are really only roughly 150 years old. Before that, there wasn't enough information about the human bo…

You are still talking about low level infrastructure. This is like studying neurons only from a cellular biology perspective and then trying to understand language acquisition in children. It is very clear from recent literature that the emergent structure and behavior of LLMs is absolutely a new research field.

Re: Embarrassingly simple self-distillation improves code generation

#212

Post by jeremyjh »

batshit_beaver wrote: post #121
danielmarkbruce wrote: post #113

Earlier quoted context omitted.

"Designed" is a bit strong. We "literally" couldn't design programs to do the interesting things LLMs can do. So we gave a giant for loop a bunch of data and a bunch of parameterized math functions and just kept updating the parameters until we got something we liked.... even on the architecture (ie, what math functions) people are just trying stuff and seeing if it works.

> We "literally" couldn't design programs to do the interesting things LLMs can do. That's a bit of an overstatement. The entire field of ML is aimed at problems where deterministic code would work just fine, but the amount of cases it would need to cover is too large to be practical (note, this has nothing to do with the impossibility of its design) AND there's a sufficient corpus of data that allows plausible enoug…

It is impossible to design even in a theoretical sense if functional requirements consider matters such as performance and energy consumption. If you have to write petabytes of code you also have to store and execute it.

Re: Embarrassingly simple self-distillation improves code generation

#213

Post by tatrions »

joshuaisaact wrote: post #199

This was a really interesting paper but there's a massive gap in what they didn't try, which is inference-time temperature changes based on the fork/lock distinction. Maybe I'll try that myself, because it feels like it could be a great source of improvements. It would be really useful to see adaptive per-token sampling as an additional decode-only baseline.

[flagged]

Re: Embarrassingly simple self-distillation improves code generation

#214

Post by krackers »

teleforce wrote: post #183

It seems that self-distillation is the way to go for LLM. Self-distillation has been shown recently as very efficient and effective back in January this year by MIT and ETH team in their Self-Distillation Fine-Tuning (SDFT) LLM system [1],[2]. This paper is also their closest competitor named On-Policy Self-Distillation in the comparison table. I hope they keep the original work real name that is Self-Distillation Fi…

Good explainer for on-policy self distillation from the authors https://x.com/siyan_zhao/status/2014372747862999382#m

Re: Embarrassingly simple self-distillation improves code generation

#215

Post by krackers »

zyklu5 wrote: post #139

Their explanation for why their idea (SSD) might work - precision-exploration conflict hypothesis - is something adaptive decoding also tries to solve. https://ai.meta.com/research/publications/adaptive-decoding-...

Seems similar to entropix?

https://github.com/xjdr-alt/entropix

Re: Embarrassingly simple self-distillation improves code generation

#216

Post by krackers »

fpgaminer wrote: post #182
andy_xor_andrew wrote: post #170

Earlier quoted context omitted.

Not really? If you read it, there is no validation, no correctness signal, no verification, none of that. They're just passing in benchmark inputs, collecting the outputs (regardless of their quality), training on those outputs, and then sweeping the decode settings (temp, topk) of the resulting model. Their conclusion is that this results in a better model than the original - even when taking into consideration the…

Not only that, they additionally ran an experiment with the training temperature turned way up (2.0) and truncation turned off such that the majority of SFT examples were incoherent (63% IIRC). Yet the model finetuned on these broken examples still improved over baseline.

Maybe this vaguely still makes sense in some way, because there is actually some useful signal purely in the model "internalizing" the behavior of its own sampler.

I don't know enough to say anything more formal, but it feels like exposing the model to its own output might help it "learn" to work with the sampler to get to a goal. I know that this is partly one of the reasons why RL is helpful, because aside from shifting the output towards a specific reward (rlvr or rlhf) it's also the only place where things are optimized at an actual "end to end sampled sequence of tokens" level instead of "next logits level" like in pretraining (which is why the highest probability suffix completion isn't necessarily simply greedy highest logit choices)

Re: Embarrassingly simple self-distillation improves code generation

#217

Post by danielmarkbruce »

astrange wrote: post #161
danielmarkbruce wrote: post #138

Earlier quoted context omitted.

Saying ML is a field where deterministic code would work just fine conveniently leaves out the difficult part - writing the actual code.... Which we haven't been able to do for most of the tasks at hand. What you are saying is fantasy nonsense.

They did not leave it out. > but the amount of cases it would need to cover is too large to be practical (note, this has nothing to do with the impossibility of its design)

It's not only too large - we can't even enumerate all the edge cases, let alone handle them. It's too difficult.

Re: Embarrassingly simple self-distillation improves code generation

#218

Post by an0malous »

dzjkb wrote: post #197
an0malous wrote: post #47

I’d like to understand AI research better and I recall some posts a while back where someone collected all the key papers that one should read, but I don’t remember enough to be able to find it. Does anyone know what I’m talking about and could link me to that post?

perhaps it was Ilya Sutskever's list? https://github.com/pageman/sutskever-30-implementations

Yes I think so, thank you!

Re: Embarrassingly simple self-distillation improves code generation

#219

Post by Tarq0n »

sgbeal wrote: post #98
jameshart wrote: post #93

Earlier quoted context omitted.

Give coding agents access to intellisense and syntax highlighting. Making coding agents spit out syntactically correct code token by token is like asking a human to code on a whiteboard.

> Give coding agents access to intellisense and syntax highlighting. i once asked an LLM if it could ingest code from an interactive session more easily if it were in appropriately-typed markdown fences and it said absolutely yes, and that the syntax highlighting fed to it that way helps it immensely. i was downright shocked that syntax highlighting was anything more than noise for them.

You can't trust what a model says about itself. It has no ability to introspect.

Re: Embarrassingly simple self-distillation improves code generation

#220

Post by jurgenburgen »

batshit_beaver wrote: post #121
danielmarkbruce wrote: post #113

Earlier quoted context omitted.

"Designed" is a bit strong. We "literally" couldn't design programs to do the interesting things LLMs can do. So we gave a giant for loop a bunch of data and a bunch of parameterized math functions and just kept updating the parameters until we got something we liked.... even on the architecture (ie, what math functions) people are just trying stuff and seeing if it works.

> We "literally" couldn't design programs to do the interesting things LLMs can do. That's a bit of an overstatement. The entire field of ML is aimed at problems where deterministic code would work just fine, but the amount of cases it would need to cover is too large to be practical (note, this has nothing to do with the impossibility of its design) AND there's a sufficient corpus of data that allows plausible enoug…

Using your logic, we don’t need quantum computers to break encryption, we could just use pen and paper.
Post reply on HN