I disagree with the author's thesis about data scarcity.
There's an infinite amount of data available in the real world. The real world is how all generally intelligent humans have been trained. Currently, LLMs have just been trained on the derived shadows (as in Plato's allegory of the cave). The grounding to base reality seems like an important missing piece. The other data type missing is the feedback: more than passively training/consuming text (and images/video), being able to push on the chair and have it push back.
Once the AI can more directly and recursively train on the real world, my guess is we'll see Sutton's bitter lesson proven out once again.
The Bitter Lesson Is Misunderstood
111–120 of 259 posts
Re: The Bitter Lesson Is Misunderstood
#112I am not an expert in AI by any means but I think I know enough about it to comment on one thing: there was an interesting paper not too long ago that showed if you train a randomly-initialized model from scratch on questions, like a bank of physics questions & answers, models will end up with much higher quality if you teach it the simple physics questions first, and then move up to more complex physics questions. T…
From my personal experience training models this is only true when the parameter count is a limiting factor. When the model is past a certain size, it doesn't really lead to much improvement to use curriculum learning. I believe most research also applies it only to small models (e.g. Phi)
Re: The Bitter Lesson Is Misunderstood
#113Earlier quoted context omitted.
This makes me think something else, though. Once we were able to reason about the physics behind the way things can move, we invented wheels. From there it's a few thousand years to steam engines and a couple hundred more years to jet planes and space travel. We may have needed a billion years of evolution from a cell swimming around to a bipedal organism. But we are no longer speed limited by evolution. Is there any…
Something that isn’t obvious when we’re talking about the invention of the wheel: we aren’t actually talking about the round shape thing, we’re actually talking about the invention of the axle which allowed mounting a stationary cart on moving wheels.
And the roadways (later, rails) on which it operates.
Meanwhile, entire civilizations in South America developed with little to no use of wheels, because the terrain was unsuited to roads.
Re: The Bitter Lesson Is Misunderstood
#114Hey folks, OOP/original author and 20-year HN lurker here — a friend just told me about this and thought I'd chime in. Reading through the comments, I think there's one key point that might be getting lost: this isn't really about whether scaling is "dead" (it's not), but rather how we continue to scale for language models at the current LM frontier — 4-8h METR tasks. Someone commented below about verifiable rewards…
> rather how we continue to scale for language models at the current LM frontier — 4-8h METR tasks
I wonder if this doesn't reify a particular business model, of creating a general model and then renting it out Saas-style (possibly adapted to largish customers).
It reminds me of the early excitement over mainframes, how their applications were limited by the rarity of access, and how vigorously those trained in those fine arts defended their superiority. They just couldn't compete with the hordes of smaller competitors getting into every niche.
It may instead be that customer data and use cases are both the most relevant and the most profitable. An AI that could adopt a small user model and track and apply user use cases would have entirely different structure, and would have demonstrable price/performance ratios.
This could mean if Apple or Google actually integrated AI into their devices, they could have a decisive advantage. Or perhaps there's a next generation of web applications that model use-cases and interactions. Indeed, Cursor and other IDE companies might have a leg up if they can drive towards modeling the context instead of just feeding it as intention to the generative LLM.
Re: The Bitter Lesson Is Misunderstood
#115The problem I am facing in my domain is that all of the data is human generated and riddled with human errors. I am not talking about typos in phone numbers, but rather fundamental errors in critical thinking, reasoning, semantic and pragmatic oversights, etc. all in long-form unstructured text. It's very much an LLM-domain problem, but converging on the existing data is like trying to converge on noise. The opportun…
This is my current drum I bang on when an uninformed stakeholder tries shoving LLMs blindly down everyone’s throats: it’s the data, stupid . Current data aggregates outside of industries wholly dependent on it (so anyone not in web advertising, GIS, or intelligence) are garbage , riddled with errors and in awful structures that are opaque to LLMs. For your AI strategy to have any chance of success, your data has to b…
I have bad news about the quality of the data in geospatial, intelligence, and advertising.
Re: The Bitter Lesson Is Misunderstood
#116In the bitter lesson essay [0], the word "data" is not mentioned a single time.
The author fundamentally misunderstands the bitter lesson.
[0] https://www.cs.utexas.edu/~eunsol/courses/data/bitter_lesson...
Re: The Bitter Lesson Is Misunderstood
#117The problem I am facing in my domain is that all of the data is human generated and riddled with human errors. I am not talking about typos in phone numbers, but rather fundamental errors in critical thinking, reasoning, semantic and pragmatic oversights, etc. all in long-form unstructured text. It's very much an LLM-domain problem, but converging on the existing data is like trying to converge on noise. The opportun…
You just need data to be directionally correct. It doesn’t have to be absolutely correct.
We still got pretty far by scraping internet data which we all know is not fully trust worthy.
Re: The Bitter Lesson Is Misunderstood
#118I am not an expert in AI by any means but I think I know enough about it to comment on one thing: there was an interesting paper not too long ago that showed if you train a randomly-initialized model from scratch on questions, like a bank of physics questions & answers, models will end up with much higher quality if you teach it the simple physics questions first, and then move up to more complex physics questions. T…
I have never heard of order of training data matter in back propagation
Re: The Bitter Lesson Is Misunderstood
#119In any field where there is a creative element, progress comes in fits and starts that are difficult to predict in advance. No one can accurately predict when we'll get the cure for cancer, for example, in spite of people working on it. But that isn't how investors operate. They want to know what they will get in exchange for giving a company a billion dollars. If you're running an AI business, you need to set expect…
Why do you assume investors don’t know about this? They know some investments follow the power law - very few of them work out but they bring most value.
The very existence of openAI and Anthropic are proof of it happening.
Imagine you were an investor and you know what you know now (creativity can’t be predicted). How would you then invest in companies? Your answer might converge on existing VC strategies.
Re: The Bitter Lesson Is Misunderstood
#120I don't think anyone has yet trained on all videos on the Internet. Plenty of petabytes left there to pretrain on, and likely just as useful once the text/audio/image pretraining is done.
It might have been trained on a select high quality of videos, say more than 10k views and only trained on its transcripts.