Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Embarrassingly simple self-distillation improves code generation

arxiv.org

Post reply on HN
Print view View original

151–160 of 227 posts

Re: Embarrassingly simple self-distillation improves code generation

#151

Post by Invictus0 »

user_7832 wrote: post #95
Invictus0 wrote: post #69

Earlier quoted context omitted.

To say we've been studying the brain for millennia is an extreme exaggeration. Modern neuroscience is only about 50 years old.

I hate to "umm, akshually" but apparently we have been studying the brain for thousands of years. I wasn't talking about purely modern neuroscience (which ironically for our topic of emergence, (often till recently/still in most places) treats the brain as the sum of its parts - be them neurons or neurotransmitters). > The earliest reference to the brain occurs in the Edwin Smith Surgical Papyrus, written in the 17th…

None of that counts as studying the brain. It's like saying rubbing sticks together to make fire counts as studying atomic energy. Those early "researchers" were hopelessly far away from even the most tangential understanding of the workings of the brain.

Re: Embarrassingly simple self-distillation improves code generation

#152

Post by crustycoder »

SEMW wrote: post #125
crustycoder wrote: post #106

"SSD improves Qwen3-30B-Instruct from 42.4% to 55.3% pass@1 on LiveCodeBench v6" I know virtually nothing about this area but my naive take is that something that means it still only passes tests around half the time doesn't seem like a particularly big jump forwards. What am I missing?

There's no shortage of benchmarks (coding or otherwise) that any competent coding model will now pass with ~100%. But no-one quotes those any more because if everyone passes them, they don't serve any useful purpose in discriminating between different models or identifying advancements So people switch to new benchmarks which either have more difficult tasks or some other artificial constraints that make them in some…

So it's the relative and not the absolute diff that matters - thanks.

Re: Embarrassingly simple self-distillation improves code generation

#153

Post by drdrek »

This is the "Factors" Bonanza in finance all over again. You get a generally useful model, then you over-fit it to some criteria and announce advancement in the field, then it performs worse in real life. New infinite academic article glitch just dropped boys!

Re: Embarrassingly simple self-distillation improves code generation

#154

Post by jameshart »

tadfisher wrote: post #118
jameshart wrote: post #93

Earlier quoted context omitted.

Give coding agents access to intellisense and syntax highlighting. Making coding agents spit out syntactically correct code token by token is like asking a human to code on a whiteboard.

Doing a tool call for autocomplete is not going to make coding agents faster. I do think there is some merit in a tool that dumps all namespaces and reachable symbols so the agent can do its own autocomplete without a round-trip.

Doesn’t need to be a tool call.

As a human coder you don’t summon intellisense. It’s just popped up into your visual field as extra input - contextual cues.

You could force intellisense state into the context vector the LLM receives.

Re: Embarrassingly simple self-distillation improves code generation

#155

Post by justinlivi »

BarryMilo wrote: post #41
spiderfarmer wrote: post #34

Earlier quoted context omitted.

I always wonder how much smaller and faster models could be if they were only trained on the latest versions of the languages I use, so for me that is PHP, SQL, HTML, JS, CSS, Dutch, English, plus tool use for my OS of choice (MacOS). Right now it feels like hammering a house onto a nail instead of the other way around.

I seem to remember that's one of the first things they tried, but the general models tended to win out. Turns out there's more to learn from all code/discussions than from just JS.

From my own empirical research, the generalized models acting as specialists outperform both the tiny models acting as specialists and the generalist models acting as generalists. It seems that if peak performance is what you're after, then having a broad model act as several specialized models is the most impactful.

Re: Embarrassingly simple self-distillation improves code generation

#156

Post by robocat »

bensyverson wrote: post #16

Really fascinating how this works; it's basically context-aware decoding. From the paper: > Code interleaves fork positions, where several continuations are genuinely plausible and may correspond to different solution approaches, with lock positions, where syntax and semantics leave little ambiguity but a low-probability distractor tail still remains… The best global decoding setting is therefore necessarily a compro…

> In other words, just like us

I think you are implying a reverse causation. They used a metaphor from us.

Re: Embarrassingly simple self-distillation improves code generation

#158

Post by darkhorse222 »

p_stuart82 wrote: post #73
khalic wrote: post #42

Earlier quoted context omitted.

Another example of the mindf@#$ these systems are: I was doing some fine tuning to a small model, take data fields and make a sentence out of it. I was running into mode collapse (basically when the AI simplifies too much and always output the same thing). I got unstuck by randomizing the field order for each row?!? At training, and now I'm thinking I should do the same at inference time...

the irony of modern software engineering: we spent decades perfecting deterministic algorithms, and now we're basically just shaking a black box and hoping the magic rocks align.

Quantum physics teaches us that at the fundamental levels of physics, reality itself is probabilistic. Probability distributions collapsing to discrete locations aligns nicely across LLMs and quantum mechanics.

Re: Embarrassingly simple self-distillation improves code generation

#159

Post by AlphaAndOmega0 »

evilduck wrote: post #127
AlphaAndOmega0 wrote: post #77

Earlier quoted context omitted.

I'm a psychiatry resident who finds LLM research fascinating because of how strongly it reminds me of our efforts to understand the human brain/mind. I dare say that in some ways, we understand LLMs better than humans, or at least the interpretability tools are now superior. Awkward place to be, but an interesting one.

To be fair to your field, that advancement seems expected, no? We can do things to LLMs that we can't ethically or practically do to humans.

I'm still impressed by the progress in interpretability, I remember being quite pessimistic that we'd achieve even what we have today (and I recall that being the consensus in ML researchers at the time). In other words, while capabilities have advanced at about the pace I expected from the GPT-2/3 days, mechanistic interpretability has advanced even faster than I'd hoped for (in some ways, we are very far from completely understanding the ways LLMs work).

Re: Embarrassingly simple self-distillation improves code generation

#160

Post by AlphaAndOmega0 »

growpdifjkl wrote: post #149
batshit_beaver wrote: post #121

Earlier quoted context omitted.

> We "literally" couldn't design programs to do the interesting things LLMs can do. That's a bit of an overstatement. The entire field of ML is aimed at problems where deterministic code would work just fine, but the amount of cases it would need to cover is too large to be practical (note, this has nothing to do with the impossibility of its design) AND there's a sufficient corpus of data that allows plausible enoug…

[flagged]

I'm a psychiatry resident who has been into ML since... at least 2017. I even contemplated leaving medicine for it in 2022 and studied for that, before realizing that I'd never become employable (because I could already tell the models were getting faster than I am).

You would be sorely mistaken to think I'm utterly uninformed about LLM-research, even if I would never dare to claim to be a domain expert.

Post reply on HN