Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

The Bitter Lesson Is Misunderstood

obviouslywrong.substack.com

Post reply on HN
Print view View original

121–130 of 259 posts

Re: The Bitter Lesson Is Misunderstood

#121

Post by Isharmla »

FloorEgg wrote: post #10
tliltocatl wrote: post #7

Earlier quoted context omitted.

That's the endgame, but on the other hand, we already have one, it's called "humanity". No reason to believe that another one would be much cheaper. Interacting with the real world is __expensive__. It's the most expensive thing of all.

Very true. Living cells are ~4-5 orders of magnitude more functional-information-dense than the most advanced chips, and there is a lot more living mass than advanced chips. But the networking potential of digital compute is a fundamentally different paradigm than living systems. The human brain is constrained in size by the width of the female pelvis. So while it's expensive, we can trade scope-constrained robustnes…

> The human brain is constrained in size by the width of the female pelvis.

https://en.wikipedia.org/wiki/Obstetrical_dilemma

While the width is constrained by bipedal locomotion.

Re: The Bitter Lesson Is Misunderstood

#122

Post by card_zero »

trescenzi wrote: post #82
programjames wrote: post #78

Earlier quoted context omitted.

It took about the same amount of time to evolve human-level intelligence as human-level mobility. Pretty much no other animal walks on two legs...

This is interesting to think about. It’s basically just birds and primates. Birds have an ancient evolutionary tree as they are dinosaurs, which did actually walk on two legs. But the gap between dinos and primates walking on two feet, I think, is tens of millions of years. So yea pretty long time.

Yes, only humans, birds, sifakas, pangolins, kangaroos, and giant ground sloths. Only those six groups of creatures, and various lizards including the Jesus lizard which is bipedal on water, just those seven groups and sometimes goats and bears.

Re: The Bitter Lesson Is Misunderstood

#123

Post by mehulashah »

I’m surprised by the argument. It’s not wrong. You need more data, but that presumes that the task is to pre-train on data. Additional compute is also useful for unearthing tacit capabilities in the models. This requires inference time scaling and post training usually on specific downstream tasks using RL. Sure that generates data, but it’s not the same as the Internet, and can be scaled.

Re: The Bitter Lesson Is Misunderstood

#124

Post by GeorgeTirebiter »

FloorEgg wrote: post #101
breuleux wrote: post #89

Earlier quoted context omitted.

You're right, we probably have different ontologies. To me an intelligent system is a system which aims to realize a goal through modelling its environment and planning actions to bring about that intended state. That's more or less what humans do and I think that's more in line with the colloquial understanding of it. There are basically two approaches to defining intelligence, I think. You can either define it in t…

If intelligence is treated as a scale, should it be measured primarily by (a) the diversity of valid actions an entity can take combined with its ability to collect and process information about its environment and predict outcomes, or (b) only by its ability to collect and process information and predict outcomes? In either case, the smallest unit of intelligence could be seen as a component of a two-field or partic…

I think it's an issue of hierarchies and the Society of Mind (Minsky). If a human touches a hot stove, or any animal's end effector, a lower-level process instantly pulls the hand/paw away from the heat. There are no doubt thousands of these 'smart body, no brain' interactions that take over in certain situations, conscious thinking not required.

Ken Goldberg shows that getting robots to operate in the real world using methods that have been successful getting LLMs to do things we consider smart -- getting huge amounts of training data -- seems unlikely. The vastness between what little data a company like Physical Intelligence has vs what GPT-5 uses is shown here: https://drive.google.com/file/d/16DzKxYvRutTN7GBflRZj57WgsFN... 84 seconds

Ken advocates plenty of Good Old-Fashioned Engineering to help close this gap, and worries that demos like Optimus actually set the field back because expectations are set too high. Like the AI researchers who were shocked by LLMs' advances, it's possible something out of left field will close this training gap for robots. I think it'll be at least 5 more years before robots will be among us as useful in-house servants. We'll see if the LLM hype has spilled over too much into the humanoid robot domain soon enough.

Re: The Bitter Lesson Is Misunderstood

#125

Post by hansvm »

geetee wrote: post #24

I don't understand why we need more data for training. Assuming we've already digitized every book, magazine, research paper, newspaper, and other forms of media, why do we need this "second internet?" Legal issues aside, don't we already have the totality of human knowledge available to us for training?

We don't have anything close to the totality of human knowledge digitized, much less in a form that LLMs can easily take advantage of. Even for easily verifiable facts powering modern industry, details like appropriate lube/speeds/etc for machining molybdenum for this or that purpose just don't exist outside of the minds of the few people who actually do it. Moreover, _most_ knowledge is similarly locked up inside a few people rather than being written down.

Even when written down, without the ability to interact with and probe the world like you did growing up it's not possible to meaningfully tell the difference between 9/11 hoaxers and everyone else save for how frequent the relative texts appear. They don't have the ability to meaningfully challenge their world model, and that makes the current breadth of written content even less useful than it might otherwise appear.

Re: The Bitter Lesson Is Misunderstood

#126

Post by Earw0rm »

kushalc wrote: post #60
FloorEgg wrote: post #56

Earlier quoted context omitted.

10+ years ago I expected we would get AI that would impact blue collar work long before AI that impacted white collar work. Not sure exactly where I got the impression, but I remember some "rising tide of AI" analogy and graphic that had artists and scientists positioned on the high ground. Recently it doesn't seem to be playing out as such. The current best LLMs I find marvelously impressive (despite their flaws), a…

Not a robotics guy, but to extent that the same fundamentals hold— I think it's a degrees of freedom question. Given the (relatively) low conditional entropy of natural language, there aren't actually that many degrees of (true) freedom. On the other hand, in the real world, there are massively more degrees of freedom both in general (3 dimensions, 6 degrees of movement per joint, M joints, continuous vs. discrete sp…

Autonomous vehicles are an interesting subset.

Even though the system rules and I/O are tightly constrained, they're still struggling to match human performance in an open-world scenario, after a gigantic R&D investment with a crystal clear path to return.

Fifteen years ago I thought that'd be a robustly solved problem by now. It's getting there, but I think I'll still need to invest in driving lessons for my teenage kids. Which is pretty annoying, honestly: expensive, dangerous for a newly qualified driver, and a massive waste of time that could be used for better things. (OK, track days and mountain passes are fun. 99% of driving is just boring, unnecessary suckage).

What's notable: AVs have vastly better sensors than humans, masses of compute, potentially 10X reaction speed. What they struggle with is nuance and complexity.

Also, AVs don't have to solve the exact same problems as a human driver. For example, parking lots: they don't need to figure out echelon parking or multi-storey lots, they can drop their passengers and drive somewhere else further away to park.

Re: The Bitter Lesson Is Misunderstood

#127

Post by pmontra »

breuleux wrote: post #89
FloorEgg wrote: post #84

Earlier quoted context omitted.

Nicely said. This all aligns with my intuition, with one caveat. I think you and I are using different definitions of intelligence. I'm bought into Karl Friston's free energy principle and think it's intelligence all the way down. There is no separating embodiment and intelligence. The LLM distinction is intelligence via symbols as opposed to embodied intelligence, which is why I really like your shadow world analogy…

You're right, we probably have different ontologies. To me an intelligent system is a system which aims to realize a goal through modelling its environment and planning actions to bring about that intended state. That's more or less what humans do and I think that's more in line with the colloquial understanding of it. There are basically two approaches to defining intelligence, I think. You can either define it in t…

> But it may then turn out to be the case that some non-intelligent systems are more effective than some intelligent systems.

That is surely the case on limited scopes. For example the non neural net chess engines are better at chess than any human.

I think that neural networks compare with human intelligence in a fair way, because we should limit their training to the number of games that human professionals can reasonably play in their life. Alphago won't be much good after playing, let's say, 10 thousand games even starting from the corpus of existing human games.

Re: The Bitter Lesson Is Misunderstood

#128

Post by taneq »

giardini wrote: post #108
breuleux wrote: post #74

Earlier quoted context omitted.

> So it makes me wonder, is embodiment (advanced robotics) 1000x harder than LLMs from an information processing perspective? Essentially, yes, but I would go further in saying that embodiment is harder than intelligence in and of itself. I would argue that intelligence is a very simple and primitive mechanism compared to the evolved animal body, and the effectiveness of our own intelligence is circumstantial. We man…

Plato's "Allegory of the cave" was uninteresting and uninformative when I first read it more than 50 years ago. It remains so today. https://en.wikipedia.org/wiki/Allegory_of_the_cave Also, other than in sculpture/dentistry/medicine I also find "ablation" to not be a particularly insightful metaphor either. Although I see ablation's application to LLMs I simply had to laugh when I first read about it: I envisioned st…

I don’t think that’s what ablation is about. It’s more like blowing parts off a bus until it ceases to be a bus. Then you find the minimal set of bus parts required to still be a bus, and that’s an indication that those parts are important to the central task of being a bus.

Re: The Bitter Lesson Is Misunderstood

#129

Post by Quarrelsome »

Chinjut wrote: post #64
Quarrelsome wrote: post #33

I really enjoyed reading this article as I found its content extremely insightful, but I fear I must whine for far too long about something entirely minor. As someone that didn't go to expensive maths club, the way people who did, talk about maths is disgraceful imho. Consider the equasion in this article: (C ~ 6 N⋅D) I can look up the symbol for "roughly equals", that was super cool and is a great part of curiousity…

What diamond symbol?

Oh its a dot. Dots, diamonds,the absense of an operator, anything is multiplication it seems. While this comment might look like a paragraph, its actually a lot of maths.

Re: The Bitter Lesson Is Misunderstood

#130

Post by datadrivenangel »

casey2 wrote: post #80

It's a boot-strapping problem. LLMs have shown that we can reproduce data that's already in the form we want, and use that data to solve novel problems. There is no shortage of data, it's just data that's in a form you want is hard to come by. You want to create a model that generates steps for a robot with a particular shape? First you have to create a robot with that shape that can walk, then create a million of th…

If a problem is worth throwing 10 million people at it, it's worth putting the problem into a deterministically solvable form.

Legal AI would be easy if we made our legal code more robust

Post reply on HN