31–40 of 53 posts
Gary Marcus saying the same things Gary Marcus has always said. It doesn’t matter what incredible things neural networks do, in his mind they’re always a dead end that will collapse any day now.
Marcus claims to have reread The Bitter Lesson. And I should say, I too have reread the text and I don't think Marcus is getting the actual original argument of here. All it say is that general purpose algorithms that scale will outperform special purpose algorithms that use information about the problem and don't scale. That's all. Everyone claiming more is hallucinating, things into this basic point. Notably general purpose algorithms aren't necessary neural nets and "X works better than Y" doesn't imply X is the best thing every.
So there's no contradiction between The Bitter Lesson and claims that LLMs have big hole and/or won't scale up to AGI.
Readers should make sure to contextualize this. We're talking about people researching AGI. Current LLM models are amazing, and will have business and societal impact. Previous ML models also had business and societal impact. None of that is contested here. The question is, what path leads to AGI, do LLM scale to AGI? That is the question being asked here, and some researchers think that it won't, it will scale super…
Readers should make sure to contextualize this. We're talking about people researching AGI. Current LLM models are amazing, and will have business and societal impact. Previous ML models also had business and societal impact. None of that is contested here. The question is, what path leads to AGI, do LLM scale to AGI? That is the question being asked here, and some researchers think that it won't, it will scale super…
But zooming out, LLMs are universal approximators, so it's trivially true that they can approximate any function that describes AGI. It's also true that logic (from logos or "word") is about reasoning constrained by language and conversations. So an LLM is the right sort of device you'd expect to achieve general intelligence.
There are arguably non-linguistic forms of intelligence, such as visual intelligence. But those also can operate on written symbols (e.g. the stream of bits from an image file).
The other relevant question is why does Gary Macus always seem so angry? It's draining reading one of his posts.
Readers should make sure to contextualize this. We're talking about people researching AGI. Current LLM models are amazing, and will have business and societal impact. Previous ML models also had business and societal impact. None of that is contested here. The question is, what path leads to AGI, do LLM scale to AGI? That is the question being asked here, and some researchers think that it won't, it will scale super…
> ... major thinker ... Thesis: language isn't a great representation, basically. I really should apply myself. Maybe I wouldn't work so hard, just shuck nonsense/pontificate.
The difference between Gary Marcus and you is the capacity to tell right from wrong. He has no problems pimping his credentials and shitting on other people's work and lying through his teeth to enrich himself. He's obviously intelligent enough to know better, but he's a singularly intellectually dishonest figure. He's a one man version of The Enquirer or Zergnet for AI, and thrives entirely on dishonest takes and di…
You could, in theory, represent that model as a linear stream of tokens, and provide it as context to an LLM directly. It would be an absurdly wasteful number of tokens, at minimum, and the attention-esque algorithm for how someone might "skim" that model given a structured query would be very different from how we skim over text, or image patches, or other things we represent in the token stream of typical multi-modal LLMs.
But could it instead be something that we provide as a tool to LLMs, and use an LLM as the reasoning system to generate structured commands that interact with it? I would wager that anyone who's read a book, drawn a map of the fantasy world within, and argued about that map's validity on the internet, would consider this a viable path.
At the end of the day, I think that the notion of a "pure LLM" is somewhat pedantic, because the very term LLM encapsulates our capability of "gluing" unstructured text to other arbitrary tools and models. Did we ever expect to tie our hands behind our back and make it so those arbitrary tools and models aren't allowed to maintain state? And if they can maintain state, then they can maintain the world model, and let the LLM apply the "bitter lesson" that compute always wins, on how to best interact with and update that state.
Readers should make sure to contextualize this. We're talking about people researching AGI. Current LLM models are amazing, and will have business and societal impact. Previous ML models also had business and societal impact. None of that is contested here. The question is, what path leads to AGI, do LLM scale to AGI? That is the question being asked here, and some researchers think that it won't, it will scale super…
The relevant question is whether Humans + LLMs are much more likely to get to AGI than humans without LLMS. And the answer is pretty obviously yes. I don't think anyone was arguing that we would get to AGI by just training on more data with exactly the same models. Practically every advance in the last few years has been building additional functionality on top of LLMs, not just scaling up the same architecture to mo…
I very naively assume the "easy" path will be similar: a very different system that's bolted on/references the foundation models, to enable the realtime/novel reasoning (outside the fixed latent space) bit that isn't possible now.
[1] https://animalcare.umich.edu/our-impact/our-impact-monitorin...
Gary Marcus saying the same things Gary Marcus has always said. It doesn’t matter what incredible things neural networks do, in his mind they’re always a dead end that will collapse any day now.
[flagged]