Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

The Bitter Lesson Is Misunderstood

obviouslywrong.substack.com

Post reply on HN
Print view View original

141–150 of 259 posts

Re: The Bitter Lesson Is Misunderstood

#141

Post by coldtea »

FloorEgg wrote: post #84
breuleux wrote: post #74

Earlier quoted context omitted.

> So it makes me wonder, is embodiment (advanced robotics) 1000x harder than LLMs from an information processing perspective? Essentially, yes, but I would go further in saying that embodiment is harder than intelligence in and of itself. I would argue that intelligence is a very simple and primitive mechanism compared to the evolved animal body, and the effectiveness of our own intelligence is circumstantial. We man…

Nicely said. This all aligns with my intuition, with one caveat. I think you and I are using different definitions of intelligence. I'm bought into Karl Friston's free energy principle and think it's intelligence all the way down. There is no separating embodiment and intelligence. The LLM distinction is intelligence via symbols as opposed to embodied intelligence, which is why I really like your shadow world analogy…

>There is n separating embodiment and intelligence.

And yet whetever IQ you have, it can't make you just play the violin without actually having embodied practice first.

Re: The Bitter Lesson Is Misunderstood

#142

Post by coldtea »

giardini wrote: post #108
breuleux wrote: post #74

Earlier quoted context omitted.

> So it makes me wonder, is embodiment (advanced robotics) 1000x harder than LLMs from an information processing perspective? Essentially, yes, but I would go further in saying that embodiment is harder than intelligence in and of itself. I would argue that intelligence is a very simple and primitive mechanism compared to the evolved animal body, and the effectiveness of our own intelligence is circumstantial. We man…

Plato's "Allegory of the cave" was uninteresting and uninformative when I first read it more than 50 years ago. It remains so today. https://en.wikipedia.org/wiki/Allegory_of_the_cave Also, other than in sculpture/dentistry/medicine I also find "ablation" to not be a particularly insightful metaphor either. Although I see ablation's application to LLMs I simply had to laugh when I first read about it: I envisioned st…

>Plato's "Allegory of the cave" was uninteresting and uninformative when I first read it more than 50 years ago. It remains so today.

Anything can be uninteresting and uninformative when one doesn't see it's interestingness or can't grok its information.

It however stood for millenia as a great device to describe multiple layers of abstractions, deeper reality vs appearance, and so on, with utility as such in countless domains.

Re: The Bitter Lesson Is Misunderstood

#143

Post by uncircle »

mbac32768 wrote: post #140
FloorEgg wrote: post #56

Earlier quoted context omitted.

10+ years ago I expected we would get AI that would impact blue collar work long before AI that impacted white collar work. Not sure exactly where I got the impression, but I remember some "rising tide of AI" analogy and graphic that had artists and scientists positioned on the high ground. Recently it doesn't seem to be playing out as such. The current best LLMs I find marvelously impressive (despite their flaws), a…

We think this because ten years ago we were all having our minds blown by DeepMind's game playing achievements and videos of dancing robots and thought this meant blue collar work would be solved imminently. But most of these solutions were more crude than they let on, and you wouldn't really know unless you were working in AI already. Watch John Carmack's recent talk at Upper Bound if you want him to see him destroy…

Thank you for this update. I vividly remember a few years ago the excitement of John Carmack announcing he was retreating into his cave to do some deep work on AGI, pushing the boundaries of the current AI research. I truly appreciate Carmack's intellectual honesty now at announcing "yeah, no, LLMs are not the way to go to recreate anything remotely close to human intelligence.". In fact, and I quote him, "we do not even have a line of sight to [the fundamentals of intelligence]."

I'm honestly relieved that one of the brightest minds in computing, with all the resources and desire to create actual super-intelligences, has had to temper hard his expectations.

Re: The Bitter Lesson Is Misunderstood

#144

Post by coldtea »

programjames wrote: post #78
breuleux wrote: post #74

Earlier quoted context omitted.

> So it makes me wonder, is embodiment (advanced robotics) 1000x harder than LLMs from an information processing perspective? Essentially, yes, but I would go further in saying that embodiment is harder than intelligence in and of itself. I would argue that intelligence is a very simple and primitive mechanism compared to the evolved animal body, and the effectiveness of our own intelligence is circumstantial. We man…

It took about the same amount of time to evolve human-level intelligence as human-level mobility. Pretty much no other animal walks on two legs...

Human-level mobility however is not much to write home about. Just one more variation of the many types seen in animals.

Human level intelligence is, otoh, qualitatively and quantitatively a bigger deal.

Re: The Bitter Lesson Is Misunderstood

#145

Post by coldtea »

noduerme wrote: post #104
breuleux wrote: post #96

Earlier quoted context omitted.

Note that wheels, steam engines, jet planes, spaceships wouldn't survive on their own in nature. Compared to natural structures, they are very simple, very straightforward. And while biological organisms are adapted to survive or thrive in complicated, ever-changing ecosystems, our machines thrive in sanitized environments. Wheels thrive on flat surfaces like roads, jet planes thrive in empty air devoid of trees, and…

Okay, but (1) we don't need to simulate physics faster than physics to make accurate-enough predictions to fly a plane, in our heads, or build a plane on paper, or to model flight in code. (2) If that's only because we've cleared out the trees and the Canada Geese and whatnot from our simplified model and "built the road" for the wheels, then necessity is also the mother of invention. "Hey, I want to fly but I keep c…

>Also, isn't that what's actually scary about AI, in a nutshell? The fact that it may radically simplify our world to facilitate e.g. paper clip production?

No, it's more about massive job losses and people left to float alone, mass increase in state control and surveillance, mass brain rot due to AI slop, and full deterioration of responsibility and services through automation and AI as a "responsibility shield".

Re: The Bitter Lesson Is Misunderstood

#146

Post by coldtea »

card_zero wrote: post #122
trescenzi wrote: post #82

Earlier quoted context omitted.

This is interesting to think about. It’s basically just birds and primates. Birds have an ancient evolutionary tree as they are dinosaurs, which did actually walk on two legs. But the gap between dinos and primates walking on two feet, I think, is tens of millions of years. So yea pretty long time.

Yes, only humans, birds, sifakas, pangolins, kangaroos, and giant ground sloths. Only those six groups of creatures, and various lizards including the Jesus lizard which is bipedal on water, just those seven groups and sometimes goats and bears.

And once every while, my cat.

Re: The Bitter Lesson Is Misunderstood

#147

Post by coldtea »

delusional wrote: post #105
programjames wrote: post #78

Earlier quoted context omitted.

It took about the same amount of time to evolve human-level intelligence as human-level mobility. Pretty much no other animal walks on two legs...

Talking about "time to evolve something" seems patently absurd and unscientific to me. All of nature evolved simultaneously. Nature didn't first make the human body and then go "that's perfect for filling the dishwasher, now to make it talk amongst itself" and then evolve intelligence. It all evolved at the same time, in conjunction. You cannot separate the mind and the body. They are the same physiological and mater…

>Nature didn't first make the human body and then go "that's perfect for filling the dishwasher, now to make it talk amongst itself" and then evolve intelligence. It all evolved at the same time, in conjunction.

Nature didn't make decisions about anything.

But it also absolutely didn't "all evolved at the same time, in conjunction" (if by that you mean all features, regarding body and intelligence, at the same rate).

>You cannot separate the mind and the body. They are the same physiological and material entity

The substrate is. Doesn't mean the nature of abstract thinking is the same as the nature of the body, in the same way the software as algorithm is not the same as hardware, even if it can only run on hardware.

But to the point: this is not about separating the "mind and the body". It's about how you can have humanoid form and all the typical human body functions for millions of years before you get human level intelligence, after many later evolution.

>Trying anyway is of course classic western canon.

It's also classic eastern canon, and several others besides.

Re: The Bitter Lesson Is Misunderstood

#148

Post by bob1029 »

mdemare wrote: post #131

Just using common sense, if we had a genius, who had tremendous reasoning ability, total recall of memories, and an unlimited lifespan and patience, and he'd read what the current LLMs have read, we'd expect quite a bit more from him than what we're getting now from LLMs. There are teenagers that win gold medals on the math olympiad - they've trained on In other words, data scarcity is not a fundamental problem, just…

I think quantization is the simplest canary.

If we can reduce the precision of the model parameters by 2~32x without much perceptible drop in performance, we are clearly dealing with something wildly inefficient.

I'm open to the possibility that over parameterization is essential as part of the training process, much like how MSAA/SSAA over sample the frame buffer to reduce information aliasing in the final scaled result (also wildly inefficient but very effective generally). However, I think for more exotic architectures (spiking / time domain) these rules don't work the same way. You can't back propagate a recurrent SNN so much of the prevailing machine learning mindset doesn't even apply.

Re: The Bitter Lesson Is Misunderstood

#149

Post by heresie-dabord »

lawrencechen wrote: post #116

In the bitter lesson essay [0], the word "data" is not mentioned a single time. The author fundamentally misunderstands the bitter lesson. [0] https://www.cs.utexas.edu/~eunsol/courses/data/bitter_lesson...

"We have to learn the bitter lesson that building in how we think we think does not work in the long run. The bitter lesson is based on the historical observations that 1) AI researchers have often tried to build knowledge into their agents, 2) this always helps in the short term, and is personally satisfying to the researcher, but 3) in the long run it plateaus and even inhibits further progress, and 4) breakthrough progress eventually arrives by an opposing approach based on scaling computation by search and learning. The eventual success is tinged with bitterness, and often incompletely digested, because it is success over a favored, human-centric approach."

Re: The Bitter Lesson Is Misunderstood

#150

Post by dr_dshiv »

jacobolus wrote: post #107
dr_dshiv wrote: post #57

Earlier quoted context omitted.

Well, if you are looking for diversity of perspective— temporal diversity may be valuable. Marsilio Ficino was hired by the Medici to translate Plato and other classical Greek works into Latin. He directly taught DaVinci, Raphael, Michelangelo, Toscanelli, etc. I mean to say that his ideas and perspectives helped spark the renaissance. Insofar as we hope for an AI renaissance and not an AI apocalypse, it might benefi…

And here you can e.g. find Ficino's correspondence translated into English, with commentary, https://archive.org/details/lettersofmarsili0000fici If you make a cursory search you can also find other translations of his works, various biographies, and a wide range of commentary and criticism by later authors. Many of Ficino's originals are also in the corpus of scanned and OCRed or recently republished texts. I'm sure…

Yes, but many of his books are not translated or ocr’d. For instance, La pestilenzia or de mysteriis.

And he is one of the most central figures of the renaissance. Less than 20% of neolatin has been digitized, let alone translated.

It is fine to question whether including neolatin, Arabic or Sanskrit in AI training will make AI better.

But for me, it is a core set of humanism that would be a shame to neglect.

Post reply on HN