The Bitter Lesson Is Misunderstood
obviouslywrong.substack.com
The Bitter Lesson Is Misunderstood
1–10 of 259 posts
Re: The Bitter Lesson Is Misunderstood
#2The problem I am facing in my domain is that all of the data is human generated and riddled with human errors. I am not talking about typos in phone numbers, but rather fundamental errors in critical thinking, reasoning, semantic and pragmatic oversights, etc. all in long-form unstructured text. It's very much an LLM-domain problem, but converging on the existing data is like trying to converge on noise.
The opportunity in the market is the gap between what people have been doing and what they are trying to do, and I have developed very specialized approaches to narrow this gap in my niche, and so far customers are loving it.
I seriously doubt that the gap could ever be closed by throwing more data and compute at it. I imagine though that the outputs of my approach could be used to train a base model to close the gap at a lower unit cost, but I am skeptical that it would be economically worth while anytime soon.
Re: The Bitter Lesson Is Misunderstood
#3I don't think Sutton's essay is misunderstood, but I agree with the OP's conclusion:
We're reaching scaling limits with transformers. The number of parameters in our largest transformers, N, is now in the order of trillions, which is the most we can apply given the total number of tokens of training data available worldwide, D, also in the order of trillions, resulting in a compute budget C = 6N × D, which is in the order of D². OpenAI and Google were the first to show these transformer "scaling laws." We cannot add more compute to a given compute budget C without increasing data D to maintain the relationship. As the OP puts it, if we want to increase the number of GPUs by 2x, we must also increase the number of parameters and training tokens by 1.41x, but... we've already run out of training tokens.
We must either (1) discover new architectures with different scaling laws, and/or (2) compute new synthetic data that can contribute to learning (akin to dreams).
Re: The Bitter Lesson Is Misunderstood
#4I don't think Sutton's essay is misunderstood, but I agree with the OP's conclusion: We're reaching scaling limits with transformers. The number of parameters in our largest transformers, N, is now in the order of trillions, which is the most we can apply given the total number of tokens of training data available worldwide, D, also in the order of trillions, resulting in a compute budget C = 6N × D, which is in the…
What about or (3) models that interact with the real world?
To be clear I also agree with your (1) and (2).
Re: The Bitter Lesson Is Misunderstood
#5I don't think Sutton's essay is misunderstood, but I agree with the OP's conclusion: We're reaching scaling limits with transformers. The number of parameters in our largest transformers, N, is now in the order of trillions, which is the most we can apply given the total number of tokens of training data available worldwide, D, also in the order of trillions, resulting in a compute budget C = 6N × D, which is in the…
What about or (3) models that interact with the real world? To be clear I also agree with your (1) and (2).
Play in the real world generates a data point every few minutes. Seems a bit slow?
Re: The Bitter Lesson Is Misunderstood
#6Earlier quoted context omitted.
What about or (3) models that interact with the real world? To be clear I also agree with your (1) and (2).
Play in the real world generates a data point every few minutes. Seems a bit slow?
What are you basing that statement on?
What exactly are you considering a "data point"?
Are you assuming one model = one agent instance?
I am pretty sure that there is more information (molecular structure) and functional information (I(Ex )) just in the room I am sitting in than all the unique, useful, digitized information on earth.
Re: The Bitter Lesson Is Misunderstood
#7I don't think Sutton's essay is misunderstood, but I agree with the OP's conclusion: We're reaching scaling limits with transformers. The number of parameters in our largest transformers, N, is now in the order of trillions, which is the most we can apply given the total number of tokens of training data available worldwide, D, also in the order of trillions, resulting in a compute budget C = 6N × D, which is in the…
What about or (3) models that interact with the real world? To be clear I also agree with your (1) and (2).
That's the endgame, but on the other hand, we already have one, it's called "humanity". No reason to believe that another one would be much cheaper. Interacting with the real world is __expensive__. It's the most expensive thing of all.
Re: The Bitter Lesson Is Misunderstood
#8Earlier quoted context omitted.
What about or (3) models that interact with the real world? To be clear I also agree with your (1) and (2).
Play in the real world generates a data point every few minutes. Seems a bit slow?
Humans experience (play in the real world) is multi modal though vision, sound, touch, pressure, muscle feedback, gravitational, etc. Its extremely rich in data. Its also not a data point its continuous stream of information. Also I would bet that humans synthesize data at the same time. Everytime we run multiple scenarios in our mind before choosing the one we execute without even thinking about it is synthesizing data. Also humans dream which is another form of data synthesizing. Allowing AI to interact with the real world is definitely a way to go.
Re: The Bitter Lesson Is Misunderstood
#9The problem I am facing in my domain is that all of the data is human generated and riddled with human errors. I am not talking about typos in phone numbers, but rather fundamental errors in critical thinking, reasoning, semantic and pragmatic oversights, etc. all in long-form unstructured text. It's very much an LLM-domain problem, but converging on the existing data is like trying to converge on noise. The opportun…
This is my current drum I bang on when an uninformed stakeholder tries shoving LLMs blindly down everyone’s throats: it’s the data, stupid. Current data aggregates outside of industries wholly dependent on it (so anyone not in web advertising, GIS, or intelligence) are garbage, riddled with errors and in awful structures that are opaque to LLMs. For your AI strategy to have any chance of success, your data has to be pristine and fresh, otherwise you’re lighting money on fire.
Throwing more compute and data at the problem won’t magically manifest AGI. To reach those lofty heights, we must first address the gaping wounds holding us back.
Re: The Bitter Lesson Is Misunderstood
#10Earlier quoted context omitted.
What about or (3) models that interact with the real world? To be clear I also agree with your (1) and (2).
That's the endgame, but on the other hand, we already have one, it's called "humanity". No reason to believe that another one would be much cheaper. Interacting with the real world is __expensive__. It's the most expensive thing of all.
Very true. Living cells are ~4-5 orders of magnitude more functional-information-dense than the most advanced chips, and there is a lot more living mass than advanced chips.
But the networking potential of digital compute is a fundamentally different paradigm than living systems. The human brain is constrained in size by the width of the female pelvis.
So while it's expensive, we can trade scope-constrained robustness (replication and redundancy at many levels of abstraction), for broader cognitive scale and fragility (data centers can't repair themselves and self-replicate).
Going to be interesting to see it all unfold... my bet is on stacking S-curves all the way.