Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

The Bitter Lesson Is Misunderstood

obviouslywrong.substack.com

Post reply on HN
Print view View original

181–190 of 259 posts

Re: The Bitter Lesson Is Misunderstood

#181

Post by petralithic »

FloorEgg wrote: post #56
kushalc wrote: post #49

Hey folks, OOP/original author and 20-year HN lurker here — a friend just told me about this and thought I'd chime in. Reading through the comments, I think there's one key point that might be getting lost: this isn't really about whether scaling is "dead" (it's not), but rather how we continue to scale for language models at the current LM frontier — 4-8h METR tasks. Someone commented below about verifiable rewards…

10+ years ago I expected we would get AI that would impact blue collar work long before AI that impacted white collar work. Not sure exactly where I got the impression, but I remember some "rising tide of AI" analogy and graphic that had artists and scientists positioned on the high ground. Recently it doesn't seem to be playing out as such. The current best LLMs I find marvelously impressive (despite their flaws), a…

> 10+ years ago I expected we would get AI that would impact blue collar work long before AI that impacted white collar work.

I'm not sure where people get this impression from, even back decades ago. Hardware is always harder than software. We had chess engines in the 20th century but a robotic hand that could move pieces? That was obviously not as easy because dealing with the physical world always has issues that dealing with the virtual doesn't.

Re: The Bitter Lesson Is Misunderstood

#182

Post by petralithic »

mdemare wrote: post #131

Just using common sense, if we had a genius, who had tremendous reasoning ability, total recall of memories, and an unlimited lifespan and patience, and he'd read what the current LLMs have read, we'd expect quite a bit more from him than what we're getting now from LLMs. There are teenagers that win gold medals on the math olympiad - they've trained on In other words, data scarcity is not a fundamental problem, just…

Humans are not tabulae rasae though. Evolution has hardwired our geniosity over millions of years.

Re: The Bitter Lesson Is Misunderstood

#183

Post by Vinnl »

TheDong wrote: post #170
energy123 wrote: post #153

Earlier quoted context omitted.

> they've trained on Somewhat apples and oranges given billions of years of evolution behind that human. GPT-5 started off as a blank slate.

This comparison is absolute nonsense. "How could a telescope see saturn, human eyes have billions of years of evolution behind them, and we only made telescopes a few hundred years ago, so they should be much weaker than eyes" "How can StockFish play chess better than a human, the human brain has had billions of years of evolution" Evolution is random, slow, and does not mean we arrive at even a local optima.

They're not saying that LLMs should be better than smart teenagers; they're saying that smart teenagers can solve some problems without needing massive amounts of data, so apparently those problems are technically solvable without those amounts of data.

Re: The Bitter Lesson Is Misunderstood

#184

Post by 1970-01-01 »

credit_guy wrote: post #87

> There is no second internet I don't know about that. LLMs have been trained mostly on text. If you add photos, audio and videos, and later even 3D games, or 3D videos, you get massively more data than the old plain text. Maybe by many orders of magnitude. And this is certainly that can improve cognition in general. Getting to AGI without audio and video, and 3D perception seems like a non-starter. And even if we th…

Yes. It's a complete oversight and wrong. This paper missed:

darknets, the deep web, Usenet, BBS, Internet2, and all other paywalled archives .

Re: The Bitter Lesson Is Misunderstood

#185

Post by hansvm »

skeezyboy wrote: post #165
energy123 wrote: post #153

Earlier quoted context omitted.

> they've trained on Somewhat apples and oranges given billions of years of evolution behind that human. GPT-5 started off as a blank slate.

[flagged]

They're born with neural hardware whose architecture has been optimized by evolution. Any choice of architecture imparts some inductive bias, making some problems easier and some problems harder to learn, and humans have the advantage that people with bad architectures (those not matching properties of the world we live in) were more likely to die or to not mate.

You're right that we don't call those inherited thought patterns memories; we call them reflexes, emotions, region-specific brain functions, etc.

Re: The Bitter Lesson Is Misunderstood

#186

Post by trescenzi »

card_zero wrote: post #122
trescenzi wrote: post #82

Earlier quoted context omitted.

This is interesting to think about. It’s basically just birds and primates. Birds have an ancient evolutionary tree as they are dinosaurs, which did actually walk on two legs. But the gap between dinos and primates walking on two feet, I think, is tens of millions of years. So yea pretty long time.

Yes, only humans, birds, sifakas, pangolins, kangaroos, and giant ground sloths. Only those six groups of creatures, and various lizards including the Jesus lizard which is bipedal on water, just those seven groups and sometimes goats and bears.

I get what you mean, that’s why the basically is there. Most, kangaroos and some lemurs in your list being the exception, do not move around primarily as bipeds. The ability to walk on two legs occasionally is different than genuinely having two legs and two arms.

Re: The Bitter Lesson Is Misunderstood

#187

Post by thfuran »

coldtea wrote: post #141
FloorEgg wrote: post #84

Earlier quoted context omitted.

Nicely said. This all aligns with my intuition, with one caveat. I think you and I are using different definitions of intelligence. I'm bought into Karl Friston's free energy principle and think it's intelligence all the way down. There is no separating embodiment and intelligence. The LLM distinction is intelligence via symbols as opposed to embodied intelligence, which is why I really like your shadow world analogy…

> There is n separating embodiment and intelligence. And yet whetever IQ you have, it can't make you just play the violin without actually having embodied practice first.

If you have sufficient motor control and dexterity, the amount of required practice should be approximately zero. Just calculate the required finger position and bow orientation, pressure, and velocity for optimal production of the desired sound and do that. That is not how humans perform physical tasks though.

Re: The Bitter Lesson Is Misunderstood

#188

Post by sindriava »

skeezyboy wrote: post #165
energy123 wrote: post #153

Earlier quoted context omitted.

> they've trained on Somewhat apples and oranges given billions of years of evolution behind that human. GPT-5 started off as a blank slate.

[flagged]

This is such a wildly misleading statement that it borders on straight up incorrect.

Re: The Bitter Lesson Is Misunderstood

#189

Post by nahuel0x »

Also to consider, how the massive datasets powering LLMs were generated? For the case of text, it was generations of humans and humans lives, experiences and interactions with the real world that coagulated into masses of text and the language itself.. not to mention the evolutionary process that made that possible. There is an history of biological computation and interaction behind what it seems to be static data.

Re: The Bitter Lesson Is Misunderstood

#190

Post by d--b »

How does the brain do it?

A baby's brain isn't wired to the entire internet. A 2-year-old has access to at most 2 years of HD video data, plus some other belly-ache and poo-smell stimuli. And a baby's brain has no replay capacity.

That's not a lot to work with.

Yet, a 2-year-old clearly thinks, is conscious, can understand and create sentences, and wants to annihilate everything just as much as Grok.

Sure you can scale data all you want. But there should be enough to work with without scaling like crazy.

Having AI know all CSS tricks out there is one thing that requires a lot of data, AGI is different.

Post reply on HN