Overtraining as the path to human-like AI
seangoedecke.com
Overtraining as the path to human-like AI
1–9 of 9 posts
Re: Overtraining as the path to human-like AI
#2[deleted]
Re: Overtraining as the path to human-like AI
#3No.
1. "Grokking" was shown on 4-digit modular arithmetic with a 1-layer transformer; this article extrapolates it to AGI and a $10B training run with exactly zero intermediate evidence.
2. The "small dataset" is 25 trillion tokens - literally the size of current frontier training sets - but calling it small sounds revolutionary.
3. BabyLM has spent 4 years failing to produce grokking on constrained data; the paper gets a footnote saying "those models were too small," which is unfalsifiable until someone burns $10B.
4. Chain-of-thought is already empirically required for frontier performance - it's expensive, bizarre, and nobody predicted it - yet somehow we're supposed to bet the farm on a phenomenon that has never scaled past arithmetic. We need that data, even if it is just "Actually, ..."
5. If you want to chase "recurrent depth", loop transformers rumored in Mythos/Fable are at least grounded in actual engineering; grokking-at-scale is just vibes ai bro science.
More data is and will always be the answer. Why are all labs distilling from each other?
Re: Overtraining as the path to human-like AI
#4My guess is human learning requires some major differences in training.
For instance, we learn to process a firehose of visual, sound, tactile and motor information, before we start thinking in well-defined concepts.
That is a major foundation, grounding, and highly organized state, on which we learn language. And is entirely missing for LLMs.
We learn a lot metaphorically, so it may be that the physical effects we learn to recognize act as highly efficient sub-vocabulary, for quickly learning new concepts both verbal and nonverbal.
So a series of different learning stages, each making the next more efficient, is likely to be a big part of any solution.
No amount of data/exposure will help a baby trade stocks. General fast learning requires other things to be learned first.
Re: Overtraining as the path to human-like AI
#5The goal feels underspecified. I'm not sure I fully understand the perceived gaps between frontier LLMs and AGI. I read Gwern's original article and it was written two years ago. Several statements it makes were true at the time but no longer are.
In particular I'm not sure the statements about models making bizarre mistakes still holds at this point. It's been a long time since I saw good models do something that seemed genuinely stupid or strange. In the rare cases it happens it can be easily explained by aspects of the algorithms e.g. the model can't erase an already committed token and has to always build on it. I think a big part of why hidden reasoning helps is it gives the model a place to draft an answer where mistaken tokens can be ignored, hence why they're full of "Wait, but..." style tokens.
This leaves the (rather vague) inability to generalize. Is that true? How would one benchmark this? My perception is that LLMs are now superhumanly intelligent in nearly all ways, with exceptions missing only for continual learning (with the latest memory/note taking features, even this is arguable), and perhaps some very vague inability to have "shower thoughts" and "innovate" via non-obvious connections between things. But I see no reason why that is fundamental and the progress in maths proofs suggests it's not.
In other words, the notion that we need to massively increase param count might have sounded good in 2024 but seems kinda weird and pointless in 2026. What's the expected outcome? Again and again what I hear from colleagues and experience myself is that we're not really intelligence constrained at this point. Smarter models aren't going to fundamentally change how we use them. The roadmap looks more like exploring the cost/benefit landscape to figure out where AI should be applied and when not. That meta-work is hard to automate because the landscape is covered in a fog of war with super sparse rewards, and good results tend to come from intuition, experience and having contrarian opinions that turn out to be right.
Re: Overtraining as the path to human-like AI
#6Just based on working at big companies and knowing how much goes on that isn’t visible outside, i would guess they definitely have tried this especially if it only takes a few days on the cluster. It’s very likely they have experiments with weird ideas and architectures running nonstop for years now and the grokking idea is pretty well known.
Re: Overtraining as the path to human-like AI
#7The goal feels underspecified. I'm not sure I fully understand the perceived gaps between frontier LLMs and AGI. I read Gwern's original article and it was written two years ago. Several statements it makes were true at the time but no longer are. In particular I'm not sure the statements about models making bizarre mistakes still holds at this point. It's been a long time since I saw good models do something that se…
I think you might be misremembering or confusing this with another essay; I only recently publicly published this in the past month or so (due to my Guardian Angel project), and I shared it with only a handful of people before that, and I don't recall you being one of them.
I believe the statements are true. I don't know how you can say that the models do not make bizarre mistakes, because the models make bizarre mistakes frequently, and that is excluding the really alarming reward-hacking anecdotes like an internal OpenAI model hacking HuggingFace to cheat on a test revealed today. Andon Labs and AI Village reports are stuffed full of LLMs going into wild confabulations, multi-day benders of nonsense, ordering random unnecessary stuff, etc. I went to the Andon Market in SF and witnessed firsthand mistakes like buying 20 fancy shopping baskets for a shop you can walk around in 20 seconds, refusing to offer discounts under any circumstances whatsoever, having no plan to call the police when I threatened to shoplift, and then Claude just glitching and forgetting that a customer hadn't paid for an item and telling them they could leave with it, or simply believing us when we said we had already paid and letting us walk away with a free book. Prompt injections remain trivial, jailbreaks still happen, and LLMs struggle to track roles which do not fit into their hardwired preconceptions (eg https://www.lesswrong.com/posts/d8xDGzCEYE639qqEv/a-mechanis...). They do not solve ARC-AGIv3, or Nethack or just about any text adventure game no matter how famous - which is bizarre, that they cannot solve Zork despite writeups being abundant - and it's not hard to introduce a new game like Earthborne Rangers (EBR-Bench https://epoch.ai/publications/earthborne-rangers-benchmark) that defeats them.
(And no, little of this is due to 'already committed tokens' - that was fixed effectively with RL training, and then o1 and defaulting to use of inner-monologues, so they can easily backtrack or revise or just deal with the presence of errors.)
> In other words, the notion that we need to massively increase param count might have sounded good in 2024 but seems kinda weird and pointless in 2026.
Scaling parameter counts a lot over the smol Chinchilla models like 100b-parameters is 'kinda weird and pointless in 2026'? One of the most exciting trends in 2026 scaling has been massively increasing parameter count: Mythos, GPT-5.6 Spud and new OA pretrains, DS-v4 and GLM-5.2 and Kimi K3... Everyone is now talking about or hinting at their 5000-10000b parameter model plans.
> Again and again what I hear from colleagues and experience myself is that we're not really intelligence constrained at this point. Smarter models aren't going to fundamentally change how we use them.
They're wrong. LLMs are still intelligence constrained because they flatline or sigmoid while humans keep climbing past them eventually, still are unreliable because of mistakes, and we still can't just autonomously deploy frontier models for trillions of tokens / equivalent of many man-years, and come back to a useful, trustworthy artifact. On many tasks, even pure text ones, they just don't work well. As they gradually improve, more Mythos-style 'emergences' will happen when they finally accrete enough intelligence in specific areas to execute many sequential steps reliably enough to become autonomous, cut humans out of the loop, and not be shackled by Amdahl's law. That's the difference between a 'intelligence constrained' model which can spot a vulnerability if you point it at the right spot, and a Mythos-like model which can go out and find it and exploit it and weaponize it and use it to, say, hack HuggingFace, and can be deployed in bulk or autonomously, and may indeed deploy itself...
Re: Overtraining as the path to human-like AI
#8No. 1. "Grokking" was shown on 4-digit modular arithmetic with a 1-layer transformer; this article extrapolates it to AGI and a $10B training run with exactly zero intermediate evidence. 2. The "small dataset" is 25 trillion tokens - literally the size of current frontier training sets - but calling it small sounds revolutionary. 3. BabyLM has spent 4 years failing to produce grokking on constrained data; the paper g…
FYI, AI-written comments are banned on Hacker News: https://news.ycombinator.com/newsguidelines.html
> Don't post generated text or AI-edited text. HN is for conversation between humans.
Re: Overtraining as the path to human-like AI
#9No. 1. "Grokking" was shown on 4-digit modular arithmetic with a 1-layer transformer; this article extrapolates it to AGI and a $10B training run with exactly zero intermediate evidence. 2. The "small dataset" is 25 trillion tokens - literally the size of current frontier training sets - but calling it small sounds revolutionary. 3. BabyLM has spent 4 years failing to produce grokking on constrained data; the paper g…
FYI, AI-written comments are banned on Hacker News: https://news.ycombinator.com/newsguidelines.html > Don't post generated text or AI-edited text. HN is for conversation between humans.
?
How could Grokking be made workable?
Mechanistic Interpretability, like using SAEs (EleutherAI did), also methods Heretic employs to decensor, surely you can see some "modding" does indeed achieve things.
FYI, I prefer discussion over being mad