[dead]
Embarrassingly simple self-distillation improves code generation
61–70 of 227 posts
Re: Embarrassingly simple self-distillation improves code generation
#62Re: Embarrassingly simple self-distillation improves code generation
#63Earlier quoted context omitted.
Learning about the emergent properties of these black boxes is not surprising, but it's also not daily. I think every new insight is worth celebrating.
Indeed. For me, it's also a good reminder that AI is here to stay as technology, that the hype and investment bubble don't actually matter (well, except to those that care about AI as investment vehicle, of which I'm not one). Even if all funding dried out today, even if all AI companies shut down tomorrow, and there are no more models being trained - we've barely begun exploring how to properly use the ones we have.…
AI has been here to stay for decades
Re: Embarrassingly simple self-distillation improves code generation
#64Re: Embarrassingly simple self-distillation improves code generation
#65Really fascinating how this works; it's basically context-aware decoding. From the paper: > Code interleaves fork positions, where several continuations are genuinely plausible and may correspond to different solution approaches, with lock positions, where syntax and semantics leave little ambiguity but a low-probability distractor tail still remains… The best global decoding setting is therefore necessarily a compro…
Another example of the mindf@#$ these systems are: I was doing some fine tuning to a small model, take data fields and make a sentence out of it. I was running into mode collapse (basically when the AI simplifies too much and always output the same thing). I got unstuck by randomizing the field order for each row?!? At training, and now I'm thinking I should do the same at inference time...
wow that's fascinating
Re: Embarrassingly simple self-distillation improves code generation
#66Earlier quoted context omitted.
Objective one should be to communicate effectively, not confuse everybody.
that disqualifies like 80% of papers lmao
Lol, you're probably not wrong. But have you ever noticed that the most important papers tend to be on the clear and readable side of things? It's as if researchers understand that being understood is important, but deemphasize that when the paper itself isn't important in the first place. (Maybe if they're only publishing to not perish, not being understood is actually a goof thing from their perspective?)
Re: Embarrassingly simple self-distillation improves code generation
#67Earlier quoted context omitted.
Indeed. For me, it's also a good reminder that AI is here to stay as technology, that the hype and investment bubble don't actually matter (well, except to those that care about AI as investment vehicle, of which I'm not one). Even if all funding dried out today, even if all AI companies shut down tomorrow, and there are no more models being trained - we've barely begun exploring how to properly use the ones we have.…
AI has been here to stay for decades
Maybe, but you couldn't tell that these days, casually scrolling this or any other tech-oriented discussion board.
Re: Embarrassingly simple self-distillation improves code generation
#68Fascinating... This feels eerily similar to sleep consolidation or synaptic pruning
I don't see much similarity? Unless you're looking at self-distillation in general and not just this use of it.
Re: Embarrassingly simple self-distillation improves code generation
#69Really fascinating how this works; it's basically context-aware decoding. From the paper: > Code interleaves fork positions, where several continuations are genuinely plausible and may correspond to different solution approaches, with lock positions, where syntax and semantics leave little ambiguity but a low-probability distractor tail still remains… The best global decoding setting is therefore necessarily a compro…
> I love that we're still learning the emergent properties of LLMs! TBH, this is (very much my opinion btw) the least surprising thing. LLMs (and especially their emergent properties) are still black boxes. Humans have been studying the human brain for millenia, and we are barely better at predicting how humans work (or for eg to what extent free will is a thing). Hell, emergent properties of traffic was not understo…
To say we've been studying the brain for millennia is an extreme exaggeration. Modern neuroscience is only about 50 years old.
Re: Embarrassingly simple self-distillation improves code generation
#70Can someone please eli5 this to a friend web developer? I read the abstract but couldn’t understand much.
if the probability mass is on a single token, its a precise answer like `1 + 1 = `
if next token predicted shares probability with other token, then there are multiple answers like `position: `
you can generate and train answers by exploring on varying the length of the code generated