Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

The Bitter Lesson

incompleteideas.net

Post reply on HN
Print view View original

41–50 of 54 posts

Re: The Bitter Lesson

#41

Post by sweezyjeezy »

colordrops wrote: post #32
sweezyjeezy wrote: post #11

This post oversimplifies the story by putting all the emphasis on compute power. Deep Blue using brute force to solve chess obviously fits this pattern, but the others? Let's take computer vision. Alex Krizhevsky et al destroyed the ImageNet competition with a neural network in 2012, kicking off the current AI hype cycle. Essentially everything in their model had been known about since the late 80s. But we also didn'…

I don't think what you are saying contradicts the text. What he's saying is that we need to put our efforts into how to design and use the tools that tackle the problem space, rather than reasoning about the problem space itself, e.g. how to use neural nets, monte carlo search, etc. That doesn't mean we just throw a for-loop at the data.

But this doesn't work either - convolutional layers in neural networks have a very specific structure, which encodes strong prior knowledge that we have about the problem space (translation invariance). If we just had multilayer perceptions, we wouldn't be talking about this right now.

Re: The Bitter Lesson

#42

Post by opticalflow »

I hate the term "AI" (even though I am CTO of a company with "AI" in its name, but since we use machine learning/DCNNs in our systems, it’s very trendy). The problem with "AI" is the "intelligence" part. Intelligence is a construct like "porn", like in the famous words of Justice Stewart about defining the latter "...but I know it when I see it". At best, it's very ambiguous -- and misleading at worst. They have been many attempts to quantitatively and qualitatively define intelligence -- none of which I find particularly satisfying and neither do any three given scientists in a room agree on a single interpretation. My problem with TFA is that it is comparing apples to oranges; deep convolutional networks are very different tools useful for a subset of problems than the ones using Bayesian inference and other statistical methods. Brute force methods like image morphology, object counting, and transforms are useful for even yet an entire Other set of problems. To say that one has displaced another is an error, in fact in most useful, modern, production systems a combination of all three is utilized, each to their purpose. To make direct comparisons between them while implying the historical decisions to use one or the other are due to Moore's Law is a false equivalence.

I clearly need my morning coffee.

Re: The Bitter Lesson

#43

Post by coldtea »

sweezyjeezy wrote: post #41
colordrops wrote: post #32

Earlier quoted context omitted.

I don't think what you are saying contradicts the text. What he's saying is that we need to put our efforts into how to design and use the tools that tackle the problem space, rather than reasoning about the problem space itself, e.g. how to use neural nets, monte carlo search, etc. That doesn't mean we just throw a for-loop at the data.

But this doesn't work either - convolutional layers in neural networks have a very specific structure, which encodes strong prior knowledge that we have about the problem space (translation invariance). If we just had multilayer perceptions, we wouldn't be talking about this right now.

>convolutional layers in neural networks have a very specific structure, which encodes strong prior knowledge that we have about the problem space

Yes. The point of the author is that it doesn't do this symbolically.

Don't get confused with the terms "brute force", "neural net", etc.

The main idea of the author is that AI that uses brute force, simpler statistical methods, NN, etc, wins over AI that tries to implement some deeper reasoning about the problem domain the way humans do (when thinking about it consciously).

Re: The Bitter Lesson

#44

Post by coldtea »

norswap wrote: post #19

I'm suspicious of hindsight bias. I'm not sure if that had written 10-20 years ago, that "learning" would figure out so predominantly. Who's to say there isn't a third such big method? Also, while the lesson fit the facts (easy in hindsight), it will hold... until it doesn't anymore. The end of Moore's law has been long heralded, and we're starting to enter this era. Progress can be made, probably, but transistors ca…

>I'm suspicious of hindsight bias.

It's the best kind of bias.

Re: The Bitter Lesson

#45

Post by beobab »

mirimir wrote: post #15

I know ~nothing about AI. But to me, this seems a great summary. And as a one-time developmental biologist, I'm struck by these observations: > One thing that should be learned from the bitter lesson is the great power of general purpose methods, of methods that continue to scale with increased computation even as the available computation becomes very great. The two methods that seem to scale arbitrarily in this way…

I was thinking the same thing. When I "play" with a new language or tool or concept, I try lots of different scenarios (search), until I can reliably predict how the new thing will work (learning).

Re: The Bitter Lesson

#46

Post by notacoward »

norswap wrote: post #19

I'm suspicious of hindsight bias. I'm not sure if that had written 10-20 years ago, that "learning" would figure out so predominantly. Who's to say there isn't a third such big method? Also, while the lesson fit the facts (easy in hindsight), it will hold... until it doesn't anymore. The end of Moore's law has been long heralded, and we're starting to enter this era. Progress can be made, probably, but transistors ca…

> I'm suspicious of hindsight bias.

As you should be, but anti-hindsight bias (or hindsight anti-bias?) is even worse. Not accusing you of that; just making a general observation. Hindsight should inform, not bias in either direction.

Re: The Bitter Lesson

#47

Post by emilfihlman »

This is a horrible post. It advocates to just throwing out research and replacing it with black boxes. Sure, they approximate (or even fully extract) the actual behaviour, but they are opaque.

I'd like to remind everyone that science is in the business of understanding, making things less opaque, less magic and engineering benefits from both.

Re: The Bitter Lesson

#48

Post by georgeecollins »

emilfihlman wrote: post #47

This is a horrible post. It advocates to just throwing out research and replacing it with black boxes. Sure, they approximate (or even fully extract) the actual behaviour, but they are opaque. I'd like to remind everyone that science is in the business of understanding, making things less opaque, less magic and engineering benefits from both.

I think you missed the point. It is saying that when we build AI systems put our understanding of a problem space into the system, we inhibit the development of a system that can create its own understanding of the problem space. He gives three very good examples of that. He also explains why people are tempted to do that: it's satisfying and initially improves the results.

Re: The Bitter Lesson

#50

Post by sgt101 »

norswap wrote: post #19

I'm suspicious of hindsight bias. I'm not sure if that had written 10-20 years ago, that "learning" would figure out so predominantly. Who's to say there isn't a third such big method? Also, while the lesson fit the facts (easy in hindsight), it will hold... until it doesn't anymore. The end of Moore's law has been long heralded, and we're starting to enter this era. Progress can be made, probably, but transistors ca…

In fact I stopped my research in supervised learning and switched to collaborative agents ~97 because I saw ML as deadended. Agents would be the thing! (hint: not so much, so far)

I think Moore's law is interesting. Technically Moore's law is about transistor density/integration, in effect it became about CPU performance and similar phenom were seen in disc and network performance. Just now we are seeing a move in general architecture away from spinning rust and towards chip based storage - ssd's and optane (or just huge DRAM) which has been much slower than I thought, but is still happening. There will be more progress as we wring out the opportunities in architecture and network devices, but overall you are right - no more Mooore's.

Also there's been a wave of progress funded by excitement - it's really hard to see how Google justified the spend on Deepmind's TPU infastructure, but they did - in contrast to a rational investment from a research council which would never have bought into Alphazero and the rest.

There's opportunity to do more - big gaps in datasets, evaluation metrics, refinement of techniques (mac nets, adversarials etc), but it's back to hardscrabble now - and I'm interested to see if this is a Warren Buffet moment. After all, you only see who's wearing shorts when the tide goes out!

Post reply on HN