Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Embarrassingly simple self-distillation improves code generation

arxiv.org

Post reply on HN
Print view View original

201–210 of 227 posts

Re: Embarrassingly simple self-distillation improves code generation

#201

Post by TeMPOraL »

darkwater wrote: post #195
59nadir wrote: post #194

Earlier quoted context omitted.

Personally I read the GP post with more emphasis on this bit: > What is (not) here to stay are the techbros who think every problem can be solved with LLMs. LLMs are in all likelyhood here to stay, but the scumbags doing business around them right now are hopefully going away eventually.

I agree on that part as well, but saying that AI will go back at what it was before ChatGPT came along is false. LLM will still be a standalone product and will be taken for granted. People will (maybe? hopefully?) eventually learn to use them properly and not generate tons of slop for the sake of using AI. Many "AI companies" will disappear from the face of Earth. But our reality has changed.

LLMs will not be just a standalone product. The models will continue to get embedded deep into software stacks, as they're already being today. For example, if you're using a relatively modern smartphone, you have a bunch of transformer models powering local inference for things like image recognition and classification, segmentation, autocomplete, typing suggestions, search suggestions, etc. If you're using Firefox and opted into it, you have local models used to e.g. summarize contents of a page when you long-click on a link. Etc.

LLMs are "little people on a chip", a new kind of component, capable of general problem-solving. They can be tuned and trimmed to specialize in specific classes of problems, at great reduction of size and compute requirements. The big models will be around as part of user interface, but small models are going to be increasingly showing up everywhere in computational paths, as we test out and try new use cases. There's so many low-hanging fruits to pick, we're still going to be seeing massive transformations in our computing experience, even if new model R&D stalled today.

Re: Embarrassingly simple self-distillation improves code generation

#202

Post by BitsAndObjects »

How is this not equivalent to training the model on the test data set? Yes it performs better at generating code for the target problems, but seemingly by becoming more tuned to the specific context of those problems (“context aware”), which suggests to me it would not generalise to real-world usage?

Re: Embarrassingly simple self-distillation improves code generation

#205

Post by elbear »

user_7832 wrote: post #40
bensyverson wrote: post #16

Really fascinating how this works; it's basically context-aware decoding. From the paper: > Code interleaves fork positions, where several continuations are genuinely plausible and may correspond to different solution approaches, with lock positions, where syntax and semantics leave little ambiguity but a low-probability distractor tail still remains… The best global decoding setting is therefore necessarily a compro…

> I love that we're still learning the emergent properties of LLMs! TBH, this is (very much my opinion btw) the least surprising thing. LLMs (and especially their emergent properties) are still black boxes. Humans have been studying the human brain for millenia, and we are barely better at predicting how humans work (or for eg to what extent free will is a thing). Hell, emergent properties of traffic was not understo…

I thought it was determined (slight pun) that free will is not a thing. I'm referring to Sapolsky's book "Determined: A Science of Life Without Free Will)" as an example.

Re: Embarrassingly simple self-distillation improves code generation

#206

Post by grumbelbart »

joshuaisaact wrote: post #199

This was a really interesting paper but there's a massive gap in what they didn't try, which is inference-time temperature changes based on the fork/lock distinction. Maybe I'll try that myself, because it feels like it could be a great source of improvements. It would be really useful to see adaptive per-token sampling as an additional decode-only baseline.

Is this some kind of calibration then? I'd expect that the probabilities automatically adjust during training, such that in "lock" mode, for example, syntax-breaking tokens have a very low probability and would not be picked even wich higher temperature.

Re: Embarrassingly simple self-distillation improves code generation

#207

Post by amelius »

guelo wrote: post #147
amelius wrote: post #53

Earlier quoted context omitted.

Studies of LLMs belong in their own field of science, just like psychology is not being studied in the physics department.

¸That field is called Machine Learning.

No that's still like putting cellular biology and psychology in the same bin.

Re: Embarrassingly simple self-distillation improves code generation

#208

Post by hellrich »

da_chicken wrote: post #193
jeremyjh wrote: post #115

Earlier quoted context omitted.

We've been studying brains a lot longer. LLMs are grown, not built. The part that is designed are the low-level architecture - but what it builds from that is incomprehensible and unplanned.

It's not that much longer, really. LLMs draw origins from, both n-gram language models (ca. 1990s) and neural networks and deep learning (ca. 2000). So we've only had really good ones maybe 6-8 years or so, but the roots of the study go back 30 years at least. Psychiatry, psychology, and neurology on the other hand, are really only roughly 150 years old. Before that, there wasn't enough information about the human bo…

Embedding „meaning“ in vector spaces goes back to 1950s structuralist linguistics and early information retrieval research, there is a nice overview in the draft for the 3rd edition of speech and language processing https://web.stanford.edu/~jurafsky/slp3/5.pdf

Re: Embarrassingly simple self-distillation improves code generation

#209

Post by thunky »

OxfordOutlander wrote: post #144

So... it's like a golfer who hits thousands of balls into an open field without ever once aiming for a hole. The relentless repetition flawlessly locks in their foundational muscle memory and basic swing mechanics, so when they finally step up to a real course, they don't have to waste a single thought on how to hold the club. Their basic swing is completely automatic - they can confidently take the creative, high-ri…

> The relentless repetition flawlessly locks in their foundational muscle memory and basic swing mechanics

If only this were true there wouldn't be an army of duffers who after a lifetime of "training" still dig a trench in front of the ball every time they play.

Re: Embarrassingly simple self-distillation improves code generation

#210

Post by sinuhe69 »

bensyverson wrote: post #16

Really fascinating how this works; it's basically context-aware decoding. From the paper: > Code interleaves fork positions, where several continuations are genuinely plausible and may correspond to different solution approaches, with lock positions, where syntax and semantics leave little ambiguity but a low-probability distractor tail still remains… The best global decoding setting is therefore necessarily a compro…

“In other words, just like us, the model needs to shift from "exploration" in "fork" mode (divergent thinking to produce a creative solution) to "precision" in "lock" mode (producing syntactically correct code).”

I’d be very cautious of the phrase 'just like us'. Not only can anthropomorphism be misleading and make us see things where none exist, it can also befuddle us, especially when we don’t know much about ourselves.

Post reply on HN