Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

The Bitter Lesson Is Misunderstood

obviouslywrong.substack.com

Post reply on HN
Print view View original

161–170 of 259 posts

Re: The Bitter Lesson Is Misunderstood

#161

Post by lambdaone »

bflesch wrote: post #152
FloorEgg wrote: post #56

Earlier quoted context omitted.

10+ years ago I expected we would get AI that would impact blue collar work long before AI that impacted white collar work. Not sure exactly where I got the impression, but I remember some "rising tide of AI" analogy and graphic that had artists and scientists positioned on the high ground. Recently it doesn't seem to be playing out as such. The current best LLMs I find marvelously impressive (despite their flaws), a…

The problem is not the robot loading the diswasher, it is the dishwasher. The dishwasher (and general kitchen electronics) industry has not innovated in a long time. My prediction is a new player will come in who vertically integrates these currently disjoint industries and product. The tableware used should be compatible with the dishwasher, the packaging of my groceries should be compatible with the cooking system.…

This exists already in the form of "ready meals" a.k.a. TV dinners. Fast food shops are already substantially mechanised; huge effots have been made to robotize cooking, but people are still cheaper to hire. It's still nowhere near the quality of home-cooked food.

Re: The Bitter Lesson Is Misunderstood

#162

Post by oblio »

Earw0rm wrote: post #113
baq wrote: post #109

Earlier quoted context omitted.

Something that isn’t obvious when we’re talking about the invention of the wheel: we aren’t actually talking about the round shape thing, we’re actually talking about the invention of the axle which allowed mounting a stationary cart on moving wheels.

And the roadways (later, rails) on which it operates. Meanwhile, entire civilizations in South America developed with little to no use of wheels, because the terrain was unsuited to roads.

It wasn't actually just terrain. It was actually availability of draft animals, climate conditions and actually most importantly... economics.

Wheeled vehicles aren't inherently better in a natural environment unless they're more efficient economically than the alternatives: pack animals, people carrying cargo, boats, etc.

South America didn't have good draft animals and lots of Africa didn't have the proper economic incentives: Sahara had bad surfaces where camels were absolutely better than carts and sub Saharan Africa had climate, terrain, tsetse flies and whatnot that made standard pack animals economically inefficient.

Humans are smart and lazy, they will do the easiest thing that let's them achieve their goals. This sometimes leads them to local maxima. That's why many "obvious" inventions took thousands of years to create (cotton gin, for example).

Re: The Bitter Lesson Is Misunderstood

#163

Post by oblio »

coldtea wrote: post #144
programjames wrote: post #78

Earlier quoted context omitted.

It took about the same amount of time to evolve human-level intelligence as human-level mobility. Pretty much no other animal walks on two legs...

Human-level mobility however is not much to write home about. Just one more variation of the many types seen in animals. Human level intelligence is, otoh, qualitatively and quantitatively a bigger deal.

I wouldn't agree completely. Being bipedal frees up the hands for, anything, really.

We're better than most animals because we have tools. We have great tools because we have hands.

Re: The Bitter Lesson Is Misunderstood

#164

Post by oblio »

imtringued wrote: post #151
programjames wrote: post #78

Earlier quoted context omitted.

It took about the same amount of time to evolve human-level intelligence as human-level mobility. Pretty much no other animal walks on two legs...

Birds? Bears whose front paws got injured? https://youtu.be/kcIkQaLJ9r8

Birds didn't develop hands, neither did bears. Also bears can't walk 100km on their hind legs, but we can.

Re: The Bitter Lesson Is Misunderstood

#165

Post by skeezyboy »

energy123 wrote: post #153
mdemare wrote: post #131

Just using common sense, if we had a genius, who had tremendous reasoning ability, total recall of memories, and an unlimited lifespan and patience, and he'd read what the current LLMs have read, we'd expect quite a bit more from him than what we're getting now from LLMs. There are teenagers that win gold medals on the math olympiad - they've trained on In other words, data scarcity is not a fundamental problem, just…

> they've trained on Somewhat apples and oranges given billions of years of evolution behind that human. GPT-5 started off as a blank slate.

[flagged]

Re: The Bitter Lesson Is Misunderstood

#166

Post by jeremyjh »

voxic11 wrote: post #103
jeremyjh wrote: post #51

Earlier quoted context omitted.

Are they actually producing new data though? This is the sort of thing I called "compression and filtering" because it seems to be new information content is not being produced, but LLMs are used to distill the information we already have. We need more raw information.

Yes this is new synthetic data which did not exist before. I encourage you to read the link.

I think we're talking past each other, I'll try once more. Suppose you train an LLM on a very small corpus of data, such as all the content of the library of congress. Then you have that LLM author new works. Then you train a new LLM on the original corpus plus this new material. Do you really think you've addressed the core issue in the SP? Can more parameters be meaningfully trained even if you add more GPU?

To me, the answer is clearly no. There is no new information content in the generated data. Its just a remix of what already exists.

Re: The Bitter Lesson Is Misunderstood

#167

Post by TheDong »

Is the data input into ChatGPT not a large enough source of new data to matter?

People are constantly inputting novel data, telling ChatGPT about mistakes it made and suggesting approaches to try, and so on.

For local tools, like claude code, it feels like there's an even bigger goldmine of data in that you can have a user ask claude code to do something, and when it fails they do it themselves... and then if only anthropic could slurp up the human-produced correct solution, that would be high quality training data.

I know paid claude-code doesn't slurp up local code, and my impression is paid ChatGPT also doesn't use input for training... but perhaps that's the next thing to compromise on in the quest for more data.

Re: The Bitter Lesson Is Misunderstood

#168

Post by skeezyboy »

charleshn wrote: post #26
jeremyjh wrote: post #20

Earlier quoted context omitted.

AlphaZero trained itself through chess games that it played with itself. Chess positions have something very close to an objective truth about the evaluation, the rules are clear and bounded. Winning is measurable. How do you achieve this for a language model? Yes, distillation is a thing but that is more about compression and filtering. Distillation does not produce new data in the same way that chess games produce…

You can have a look at the DeepSeek paper, in particular section "2.2 DeepSeek-R1-Zero: Reinforcement Learning on the Base Mode". But generally the idea is that it's, you need some notion of reward, verifiers etc. Works really well for maths, algorithms, amd many things actually. See also this very short essay/introduction: https://www.jasonwei.net/blog/asymmetry-of-verification-and-... That's why we have IMO gold le…

> But generally the idea is that it's, you need some notion of reward, verifiers etc.

i dont think youre getting the point hes making.

Re: The Bitter Lesson Is Misunderstood

#169

Post by imtringued »

FloorEgg wrote: post #65
airstrike wrote: post #18

Earlier quoted context omitted.

> Living cells are ~4-5 orders of magnitude more functional-information-dense than the most advanced chips, and there is a lot more living mass than advanced chips. I believe you but I would love to know where this number came from just so I can read more about it

It's napkin math so take it with a pinch of salt, but I am calculating the information stored in genome, assuming 2 bits per base pair, reducing to estimated 88% coding fraction to get the functional bits, and then dividing by cell volume. Did this for a few different types of cells and then averaged the result to around 1–10 Mbit/μm³ # If there are any bioinformaticians around please come eviscerate or confirm this…

You are comparing the fastest writable memory available (SRAM) vs biological non-volatile memory that is essentially read only. Samsung's 280 layer NAND reaches 28,5 Gbit per mm^2. I don't know how you would convert that to a volume, but if we simply multiply by 1000x for simplicity, it would be much closer to 0.19 Mbit/μm³, but even then you have to remember that NAND flash is still writable at pretty high speeds.

Re: The Bitter Lesson Is Misunderstood

#170

Post by TheDong »

energy123 wrote: post #153
mdemare wrote: post #131

Just using common sense, if we had a genius, who had tremendous reasoning ability, total recall of memories, and an unlimited lifespan and patience, and he'd read what the current LLMs have read, we'd expect quite a bit more from him than what we're getting now from LLMs. There are teenagers that win gold medals on the math olympiad - they've trained on In other words, data scarcity is not a fundamental problem, just…

> they've trained on Somewhat apples and oranges given billions of years of evolution behind that human. GPT-5 started off as a blank slate.

This comparison is absolute nonsense.

"How could a telescope see saturn, human eyes have billions of years of evolution behind them, and we only made telescopes a few hundred years ago, so they should be much weaker than eyes"

"How can StockFish play chess better than a human, the human brain has had billions of years of evolution"

Evolution is random, slow, and does not mean we arrive at even a local optima.

Post reply on HN