31–40 of 227 posts
> simple self-distillation (SSD): Sorry apple, SSD is already taken, you can't use that acronym.
Haven't read the paper yet, but it is interesting how seemingly simple many breakthroughs in ML are. Even transformers are like that. Maybe it's hindsight bias. I suppose we just don't have a deeper underlying theory to lean on and help us 'design' anything.
A lot of discoveries are like that. In fact, simplicity is often the hallmark of correctness, and complexity is often a sign that our understanding is incomplete and we’re still stumbling towards the right model. Not always, but often. It’s been a good rule of thumb in my programming career.
I often find, if I've got a complicated solution, it’s because I haven’t fully examined the problem.
After TurboQuant and Gemma 4, came across the following video[0] running Gemma on local machine at 50 token/second. That already looks like Sonnet 3x and 4 level capabilities to me where the model in question (Gemma 4) set ups whole python project with a UI and installs python libraries using uv etc. Add this Simple Self Distillation to the picture and by 2028 I see cheaper coding model providers with much more gener…
Right now it feels like hammering a house onto a nail instead of the other way around.
Earlier quoted context omitted.
The phrase embarrassingly parallel has a history in computer science. Many computer science paper titles allude to past titles in other CS papers. Calling it “cringe worthy” is unnecessarily mean. There is context and history you don’t understand.
"Embarrassingly" considered harmful?
Shouldn't a scientific paper be using metric units (like 30T) rather than 30B. There are two distinct billions. https://en.wikipedia.org/wiki/Billion
Really fascinating how this works; it's basically context-aware decoding. From the paper: > Code interleaves fork positions, where several continuations are genuinely plausible and may correspond to different solution approaches, with lock positions, where syntax and semantics leave little ambiguity but a low-probability distractor tail still remains… The best global decoding setting is therefore necessarily a compro…
Really fascinating how this works; it's basically context-aware decoding. From the paper: > Code interleaves fork positions, where several continuations are genuinely plausible and may correspond to different solution approaches, with lock positions, where syntax and semantics leave little ambiguity but a low-probability distractor tail still remains… The best global decoding setting is therefore necessarily a compro…
TBH, this is (very much my opinion btw) the least surprising thing. LLMs (and especially their emergent properties) are still black boxes. Humans have been studying the human brain for millenia, and we are barely better at predicting how humans work (or for eg to what extent free will is a thing). Hell, emergent properties of traffic was not understood or properly given attention to, even when a researcher, as a driver, knows what a driver does. Right now, on the front page, is this post:
> 14. Claude Code Found a Linux Vulnerability Hidden for 23 Years (mtlynch.io)
So it's pretty cool we're learning new things about LLMs, sure, but it's barely surprising that we're still learning it.
(Sorry, mini grumpy man rant over. I just wish we knew more of the world but I know that's not realistic.)