Game over for pure LLMs. Even Rich Sutton has gotten off the bus

The difference between Gary Marcus and you is the capacity to tell right from wrong.

He has no problems pimping his credentials and shitting on other people's work and lying through his teeth to enrich himself. He's obviously intelligent enough to know better, but he's a singularly intellectually dishonest figure.

He's a one man version of The Enquirer or Zergnet for AI, and thrives entirely on dishonest takes and divisive commentary, subsiding on pure clickbait. There is absolutely no reason to regard anything he says with any level of seriousness or credulity, he's an unprincipled jackass cashing out unearned regard by grifting and shilling, loudly.

If you must, here's an archived link, don't reward him with clicks.

https://archive.is/L09dP

He really shouldn't end up on the top ten of HN, let alone the front page. It's like an SEO hack boosting some guy proudly documenting pictures of his bowel movements.

People can laugh at Gary Marcus all they want, but there’s one aspect that people don’t understand.

If you have a high conviction belief that is countervailing to the mainstream, you suffer a great deal. Even the most average conversation with a “mainstream believer” can turn into a judgment fest. Sometimes people stop talking to you mid-conversation. Investors quietly remove you from their lead lists. Candidates watch your talks and go dark on you. People with no technical expertise lecture at you.

Yet, inevitably, a fraction of such people carry forward. They don’t shut up. And they are the spoon that stirs the pot of science.

It’s totally normal and inevitable that people start to take victory laps at even the smallest indication in such situations. It doesn’t mean they’re right. It is just not something worth criticizing.

Demonstrating that Rich Sutton was never really on the 'LLM bus' in the first place. Note the remarkable absence from the essay of language models & large language models from that essay despite BERT and GPT-2 and 'unreasonable effectiveness of data' etc. He only briefly mentions speech recognition. (Note also Sutton's general absence from LLM research, the Edmund Plan or switch from DeepMind to Keen Technologies as DeepMind was forced into LLM-centric research, and his published research since 2019's emphasis on small models and trying to fix their pathologies like catastrophic forgetting.)

> The bitter lesson is based on the historical observations that 1) AI researchers have often tried to build knowledge into their agents, 2) this always helps in the short term, and is personally satisfying to the researcher, but 3) in the long run it plateaus and even inhibits further progress, and 4) breakthrough progress eventually arrives by an opposing approach based on scaling computation by search and learning. The eventual success is tinged with bitterness, and often incompletely digested, because it is success over a favored, human-centric approach.

You could easily seem most LLM work as a dead end because it is about 'building knowledge into your agents' (eg. by paying data labelers billions of dollars total to supplement your scrapes), and not about 'search' (still a major open problem for LLMs - o1-style serial reasoning traces are obviously inadequate) or 'learning' (LLMs depend so heavily on the knowledge already encoded in so much data for them).

“On top of LLMs” is exactly not “pure LLMs”, though, and it’s also not clear if TTC will end up supporting the bitter lesson.

No. There is a difference between being a contrarian who is making a bet and acting on their different viewpoint, and someone who is simply a debbie downer and wants to say something negative all the time with no skin in the game. Marcus is the latter.

It's easy to be a faux contrarian that just always says we're in a bubble or x is overhyped. Everyone knows that, it's the nature of markets and not an insight. The only value is having some actual insight into where and how things stop and where there is some upside. Otherwise you're just a jealous loser.