Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

The Bitter Lesson Is Misunderstood

obviouslywrong.substack.com

Post reply on HN
Print view View original

231–240 of 259 posts

Re: The Bitter Lesson Is Misunderstood

#231

Post by bflesch »

criddell wrote: post #220
bflesch wrote: post #152

Earlier quoted context omitted.

The problem is not the robot loading the diswasher, it is the dishwasher. The dishwasher (and general kitchen electronics) industry has not innovated in a long time. My prediction is a new player will come in who vertically integrates these currently disjoint industries and product. The tableware used should be compatible with the dishwasher, the packaging of my groceries should be compatible with the cooking system.…

> the whole notion of putting one room of your apartment full with random electronics just to cook a meal once in a blue moon is deeply inefficient You don't use your kitchen? After the rooms we sleep in, the kitchen is probably the most used space in my home. We are planning an upcoming renovation of our home and the kitchen is where we plan on spending the most money. > The tableware used should be compatible with…

Yes, of course I use it a lot. It is a great hobby. But only use it because it is kind of forced upon us. It's just so inefficient nowadays. Cooking used to be for the whole homestead or for the large family. Now it is mostly only for the immediate family. All the machines are not utilized properly. When people discussed car sharing it was exactly the same argument and I feel it also applies to kitchens.

With the "tableware" argument I meant something like a standardized (magnetic?) adapter for grabbing plates, forks and knives so they can easily be moved by machines/robots.

I feel a company like Ikea is perfectly set up to make this idea a reality, but they'll never do so because they make much more money when every single household buys all these appliances and items for their own kitchen.

Just from the perspective of a single household in a densely populated city I think it'd be nice to have freshly cooked, reproducibly prepared meals with high-quality ingredients available to me. Like an automated soup kitchen with cleanup. Without all the layers of plastic wrapping needed to move produce from large-scale distributors into single-household fridges and so on.

Re: The Bitter Lesson Is Misunderstood

#232

Post by airstrike »

FloorEgg wrote: post #226
imtringued wrote: post #169

Earlier quoted context omitted.

You are comparing the fastest writable memory available (SRAM) vs biological non-volatile memory that is essentially read only. Samsung's 280 layer NAND reaches 28,5 Gbit per mm^2. I don't know how you would convert that to a volume, but if we simply multiply by 1000x for simplicity, it would be much closer to 0.19 Mbit/μm³, but even then you have to remember that NAND flash is still writable at pretty high speeds.

This is true, I was only comparing functional information density (unique functional genome bps), not read/write speeds. I was also taking the information from a genome and then dividing it by the volume of a cell, but there are many instances of the genome in a cell. I didn't count all instances because they aren't unique. There's a lot to unpack with this comparison and my approximation was crude, but the more I've…

If you find that illusive paper, please post it here!

Re: The Bitter Lesson Is Misunderstood

#233

Post by jeremyjh »

voxic11 wrote: post #196
jeremyjh wrote: post #166

Earlier quoted context omitted.

I think we're talking past each other, I'll try once more. Suppose you train an LLM on a very small corpus of data, such as all the content of the library of congress. Then you have that LLM author new works. Then you train a new LLM on the original corpus plus this new material. Do you really think you've addressed the core issue in the SP? Can more parameters be meaningfully trained even if you add more GPU? To me,…

Yes if you have some way to verify the quality of the new works and you only include the high quality works in the new LLM's training set.

But you don't have a way to do that at scale, other than feed it to another LLM that is trained on that exact same limited corpus. There is no new information being added into the system in loops like that. New information means new measurements, new proofs, new signal or media streams from cameras, new curation/rating data, new books or papers etc.

Re: The Bitter Lesson Is Misunderstood

#234

Post by Quarrelsome »

Chinjut wrote: post #211
Quarrelsome wrote: post #129

Earlier quoted context omitted.

Oh its a dot. Dots, diamonds,the absense of an operator, anything is multiplication it seems. While this comment might look like a paragraph, its actually a lot of maths.

But diamonds don't denote multiplication. That never happened. That was just you misreading.

maybe it was just a fancy dot. But it looked like ◆ on the page.

> That never happened

Bit harsh, I don't see the need for gaslighting. Sure I might be losing my mind but I specifically remember it because it took me so long to find a symbol that matched it online.

Re: The Bitter Lesson Is Misunderstood

#235

Post by Quarrelsome »

ghkbrew wrote: post #204
Quarrelsome wrote: post #132

Earlier quoted context omitted.

Its good we all learned this convention. Thanks for teaching to it to me though. To clarify, if it read: C ~ X N⋅D you'd be as confused as me? Its because its a number it has special implied mechanics where we can skip operators because its "obvious".

Well no actually it'd still clear to me that they mean the the multiplication of 3 different variables X, N, and D. I don't think of it as eliding obvious operators. Rather in mathematics juxtaposition is used as an operator to represent multiplication. You would never elide an addition operator. So X next to D still means multiplication as long as you can tell that X and D are separate entities. I would wonder why t…

> Well no actually it'd still clear to me that they mean the the multiplication of 3 different variables X, N, and D.

Sure, but its not clear to me. I'm just cross about implied convention in maths.

Re: The Bitter Lesson Is Misunderstood

#236

Post by EZ-Cheeze »

WesolyKubeczek wrote: post #224
EZ-Cheeze wrote: post #171

The AI companies won't run out of data to train on. Almost every user interaction is a significant source of data. Chains of interactions are even more significant, especially the longer and more sophisticated they are. Yesterday I was given A/B tests from both GPT5-Thinking and Gemini 2.5 Pro, something neither of then had done before. OpenAI also just acquired Statsig for $1.1 billion. Statsig does A/B testing and…

> it bootstrapped something that we all love talking to and discussing ANYTHING with. We all? Speak for yourself, dude

Hyperbole, I meant a huge number of people, not ALL people.

I'm into advaita vedanta, priority cosmopsychism, open individualism

Re: The Bitter Lesson Is Misunderstood

#237

Post by qcnguy »

jebarker wrote: post #214
bob1029 wrote: post #148

Earlier quoted context omitted.

I think quantization is the simplest canary. If we can reduce the precision of the model parameters by 2~32x without much perceptible drop in performance, we are clearly dealing with something wildly inefficient. I'm open to the possibility that over parameterization is essential as part of the training process, much like how MSAA/SSAA over sample the frame buffer to reduce information aliasing in the final scaled re…

It’s not clear that the inefficiency of the current paradigm is in the neural net architectures. It seems just as likely that it’s in the training objective.

Right. The objective is "correctly predict the entire training set", where that training set contains literally everything. So the objective becomes to speak every human language, every programming language, to understand every topic, to master every weird sub-genre of culture. That's an inherently very inefficient training objective if you just want an AI that can do some specific tasks. It's the whole insight behind models specific to summarization, text extraction, patch merging etc.

And don't forget the noise. If you look at the Anthropic papers it's clear from the examples they give that the dataset is still incredibly noisy even after extensive cleaning efforts. A lot of those parameters are being wasted trying to predict garbage outputs from HTML scraping gone wrong.

Re: The Bitter Lesson Is Misunderstood

#238

Post by qcnguy »

credit_guy wrote: post #87

> There is no second internet I don't know about that. LLMs have been trained mostly on text. If you add photos, audio and videos, and later even 3D games, or 3D videos, you get massively more data than the old plain text. Maybe by many orders of magnitude. And this is certainly that can improve cognition in general. Getting to AGI without audio and video, and 3D perception seems like a non-starter. And even if we th…

That's been done already for years. OpenAI were training on bulk AI transcribed YouTube vids already in the GPT-4 era. Modern models are all multi-modal and cotrained on audio and image tokens together with text.

The AI companies are not only out of such data but their access to it is shrinking as the people who control the hosting sites wall them off (like YouTube).

Re: The Bitter Lesson Is Misunderstood

#239

Post by qcnguy »

decimalenough wrote: post #71
geetee wrote: post #24

I don't understand why we need more data for training. Assuming we've already digitized every book, magazine, research paper, newspaper, and other forms of media, why do we need this "second internet?" Legal issues aside, don't we already have the totality of human knowledge available to us for training?

The goal/theory behind the LLM investment explosion is that we can get to AGI by feeding them all the data. And to be clear, by AGI I don't mean "superhuman singularity", just "intelligent enough to replace most humans" (and, by extension, hoover up all the money we're spending on their salaries today). But if we've already fed them all the data, and we don't have AGI (which we manifestly don't), then there's no way…

That's only true if model intelligence is the limiting factor. I don't believe it is. Right now the limiting factor for AI impact is mostly the effort required to wire it up to all the systems we want it to automate.

Re: The Bitter Lesson Is Misunderstood

#240

Post by scotty79 »

jebarker wrote: post #227
scotty79 wrote: post #42

Earlier quoted context omitted.

Simple, you just need to turn language into a game. You make models talk to each other, create puzzles for each other's to solve, ask each other to make cases and evaluate how well they were made. Will some of it look like ramblings of pre-scientific philosophers? (or modern ones because philosophy never progressed after science left it in the dust) Sure! But human culture was once there too. And we pulled ourselves…

I feel like you’re glossing over some very thorny details that it’s not obvious we can solve. For example, if you just get two LLMs setting each other puzzles and scoring the others solutions how do you stop this just collapsing into nonsense? I.e. where does the source of actual truth come from for the puzzles?

> I feel like you’re glossing over some very thorny details that it’s not obvious we can solve.

Yeah. I tried to be funny. It's not that easy. However AI people already started doing it and AI gains perhaps of the last year come mostly from this approach.

> For example, if you just get two LLMs setting each other puzzles and scoring the others solutions how do you stop this just collapsing into nonsense?

That's the trillion dollar question. I wonder how people are doing it. Maybe through economy? You ultimately need to sell your ramblings to somebody to sustain yourself. If you can't, you starve.

Maybe that's enough for AI as well? Companies with AIs that descended into nonsense won't have anymore money to train them further. Maybe companies will need to set up their internal ecosystems of competing AI training organizations and split the budget based on how useful they are becoming?

Phrasing this in a terminology of "truth" is probably counterproductive because there's no truth. There's only what sells. If you have customers in manufacturing probably things that sell will coincide with some physical truths, but this is emergent, not the goal or even part of the process or acquiring capabilities.

Post reply on HN