Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

The Bitter Lesson (2019) [pdf]

cs.utexas.edu

Post reply on HN
Print view View original

11–20 of 62 posts

Re: The Bitter Lesson (2019) [pdf]

#11

Post by PoignardAzur »

v9v wrote: post #6

Also see A Better Lesson: https://rodneybrooks.com/a-better-lesson/

Ehh, I'm more on Sutton's side so far.

Brook's post goes over the classics (Moore's law is ending, curating a dataset requires human intervention, etc) and posits that making a huge model won't be a competitive strategy for long because it gets to expensive to train and use.

It's a bit early to tell, but so far that hasn't materialized. OpenAI got state-of-the-art results with GPT-4, AFAIK by sticking for very-super-big models together. Open source experiments with LLAMA show you can still get good results with heavy quantization. Distillation hasn't be too explored by mainstream projects, but I bet there's lots of potential there too.

Right now the winning strategy looks to be "go really big, then figure out how to go small".

Re: The Bitter Lesson (2019) [pdf]

#12

Post by Xcelerate »

> The second general point to be learned from the bitter lesson is that the actual contents of minds are tremendously, irredeemably complex; we should stop trying to find simple ways to think about the contents of minds, such as simple ways to think about space, objects, multiple agents, or symmetries.

I agree and think “ML explainability” efforts are doomed to fail as ML becomes increasingly more effective. There is no a priori reason that the human brain should be capable of intuitively grokking sufficiently advanced general learners. We can invent them and improve them, but saying that we will be able to understand what the myriad matrix multiplications are “doing” will be like saying we understand the human brain because we can model the physics of its constituent atoms. The emergent complexity is too high for us to make any sense of it.

Re: The Bitter Lesson (2019) [pdf]

#13

Post by reacweb »

isaacfrond wrote: post #7

I think the Go example is facetious. For the longest time, Go simply was not amenable to search. It is only because of breakthrough in theory that Go became amendable to search. And, it is hardly a straightforward search mind you. Two self learning networks are involved. A first to guide the expansion of a Monte Carlo based tree search, a second to evaluate the nodes in the tree. You can hardly blame a researcher in…

Go has been described as the drosophila of AI. Once a program will know how to play go, we'll be able to make more general artificial intelligences. This is exactly what has happened with alphago and chatgpt. We're not repeating the same thing that happened with chess, but we are past the singularity. Recent advances in AI are lighting up the very nature of intelligence. Our memory and our brain are a machine for predicting the future. It's very similar to what recent models do. Alphagozero has rediscovered the history of chess openings in the same order as humans. We are no longer discovering simple new algorithms supported by machine advances, but architecting predictive systems based on guided learning. When we developed the algorithm for playing chess, the possibilities for generalization were limited. With these new architectures, new possibilities are appearing every day. IMHO, the predictions were true: since alphago, we've entered a new world.

Re: The Bitter Lesson (2019) [pdf]

#14

Post by davegirdywww »

psychoslave wrote: post #9
niemandhier wrote: post #8

I think the Bitter Lessons do not consider two important points with regards to neural nets : 1. Not all nets work for all problems, those that work tend to have the right inductive biases. We discovered the architectures partially by trial an error, nevertheless they work because of encoded prior information. 2. Data and computation are bounded. GPT4 was basically trained on all text, further advancements probably n…

What do you mean with more insights? If they are not represented as some data, how could they ever influence any computational model?

Nobody said anything about insights.

Re: The Bitter Lesson (2019) [pdf]

#15

Post by PartiallyTyped »

PoignardAzur wrote: post #11
v9v wrote: post #6

Also see A Better Lesson: https://rodneybrooks.com/a-better-lesson/

Ehh, I'm more on Sutton's side so far. Brook's post goes over the classics (Moore's law is ending, curating a dataset requires human intervention, etc) and posits that making a huge model won't be a competitive strategy for long because it gets to expensive to train and use. It's a bit early to tell, but so far that hasn't materialized. OpenAI got state-of-the-art results with GPT-4, AFAIK by sticking for very-super-…

Going really big and figuring out how to go small is also working; see Lottery Tickets, ensembles, distillation, and so on.

Re: The Bitter Lesson (2019) [pdf]

#16

Post by YeGoblynQueenne »

reacweb wrote: post #13
isaacfrond wrote: post #7

I think the Go example is facetious. For the longest time, Go simply was not amenable to search. It is only because of breakthrough in theory that Go became amendable to search. And, it is hardly a straightforward search mind you. Two self learning networks are involved. A first to guide the expansion of a Monte Carlo based tree search, a second to evaluate the nodes in the tree. You can hardly blame a researcher in…

Go has been described as the drosophila of AI. Once a program will know how to play go, we'll be able to make more general artificial intelligences. This is exactly what has happened with alphago and chatgpt. We're not repeating the same thing that happened with chess, but we are past the singularity. Recent advances in AI are lighting up the very nature of intelligence. Our memory and our brain are a machine for pre…

>> Go has been described as the drosophila of AI.

John McCarthy described chess as "the drosophila of AI":

http://jmc.stanford.edu/articles/drosophila/drosophila.pdf

Who was it that described Go as the drosophila of AI?

Re: The Bitter Lesson (2019) [pdf]

#17

Post by petesergeant »

fnbr wrote: post #2

This is the original page, I’d link to this: http://www.incompleteideas.net/IncIdeas/BitterLesson.html u/dang, swap links if you see this?

Yah, entirely unclear why this has been submitted as a browser-saved PDF of the original HTML, when the original HTML is right there!

Re: The Bitter Lesson (2019) [pdf]

#18

Post by rstuart4133 »

I don't doubt the taste was bitter at the time, but it was never going to be forever. In 2018 models had 100e6 parameters, and in 2013 GPT4 has (wild guess) 2e12 parameters [0]. Conservatively that's 500% per year. At it's peak, hardware never improved at that rate and has since slowed down dramatically.

So it was always going to end, and further advancement was always going to revert being driven by a neural net of some sort. In fact, it looks to me we are already at that point.

The interesting question is what neural net will end up driving it.

[0] https://www.qualcomm.com/news/onq/2023/07/generative-ai-tren...

Re: The Bitter Lesson (2019) [pdf]

#19

Post by isaacfrond »

YeGoblynQueenne wrote: post #16
reacweb wrote: post #13

Earlier quoted context omitted.

Go has been described as the drosophila of AI. Once a program will know how to play go, we'll be able to make more general artificial intelligences. This is exactly what has happened with alphago and chatgpt. We're not repeating the same thing that happened with chess, but we are past the singularity. Recent advances in AI are lighting up the very nature of intelligence. Our memory and our brain are a machine for pre…

>> Go has been described as the drosophila of AI. John McCarthy described chess as "the drosophila of AI": http://jmc.stanford.edu/articles/drosophila/drosophila.pdf Who was it that described Go as the drosophila of AI?

It was John McCarthy, in the very paper that you cite!

See section 7:

As a fourth Drosophila I would like to mention the research on Computer Go.

Re: The Bitter Lesson (2019) [pdf]

#20

Post by currymj »

isaacfrond wrote: post #7

I think the Go example is facetious. For the longest time, Go simply was not amenable to search. It is only because of breakthrough in theory that Go became amendable to search. And, it is hardly a straightforward search mind you. Two self learning networks are involved. A first to guide the expansion of a Monte Carlo based tree search, a second to evaluate the nodes in the tree. You can hardly blame a researcher in…

i don't know. it's mysterious that nobody even came up with the algorithms for decades, which often do still work well on toy examples. MCTS is the kind of thing people definitely could have thought of at any point since 1950; and neural network value functions for board games had already succeeded with TD-GAMMON.

this is even more striking in poker, where counterfactual regret minimization wasn't invented until 2007, despite being a relatively simple algorithm to describe and all the essential intellectual building blocks being known since the 1960s.

von Neumann had invented extensive form games with the express idea of modeling poker; there were researchers in the 1950s/60s (Hannan, Blackwell) working out the core ideas of regret minimization. One innovation (the sequence form representation of game strategies) was only known in the Soviet Union in the 1960s and not propagated to the West. But this was independently rediscovered in the early 1990s and it still took another 15 years for CFR to be developed.

Post reply on HN