Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

The Bitter Lesson

incompleteideas.net

Post reply on HN
Print view View original

51–54 of 54 posts

Re: The Bitter Lesson

#51

Post by alew1 »

coldtea wrote: post #43
sweezyjeezy wrote: post #41

Earlier quoted context omitted.

But this doesn't work either - convolutional layers in neural networks have a very specific structure, which encodes strong prior knowledge that we have about the problem space (translation invariance). If we just had multilayer perceptions, we wouldn't be talking about this right now.

> convolutional layers in neural networks have a very specific structure, which encodes strong prior knowledge that we have about the problem space Yes. The point of the author is that it doesn't do this symbolically. Don't get confused with the terms "brute force", "neural net", etc. The main idea of the author is that AI that uses brute force, simpler statistical methods, NN, etc, wins over AI that tries to impleme…

Hmm, I'm not sure I see the difference. Why is it not "symbolic"? The symbols that construct the neural network are what encodes translation invariance -- not some vector of reals.

Re: The Bitter Lesson

#52

Post by mirimir »

beobab wrote: post #45
mirimir wrote: post #15

I know ~nothing about AI. But to me, this seems a great summary. And as a one-time developmental biologist, I'm struck by these observations: > One thing that should be learned from the bitter lesson is the great power of general purpose methods, of methods that continue to scale with increased computation even as the available computation becomes very great. The two methods that seem to scale arbitrarily in this way…

I was thinking the same thing. When I "play" with a new language or tool or concept, I try lots of different scenarios (search), until I can reliably predict how the new thing will work (learning).

That's pretty much how our brains develop. Neurons are vastly overproduced, during fetal development through the first few years. Ones that make useful connections, and do useful stuff, survive. And the rest die.

Also, as in evolution, ~random variations occur during neuronal proliferation, so there's also selection on epigenetic differences. The same sort of process occurs in the immune system.

In this way, organisms can transcend limitations of their genetic sequences. There's learning at levels of both structure and function.

Re: The Bitter Lesson

#53

Post by coldtea »

alew1 wrote: post #51
coldtea wrote: post #43

Earlier quoted context omitted.

> convolutional layers in neural networks have a very specific structure, which encodes strong prior knowledge that we have about the problem space Yes. The point of the author is that it doesn't do this symbolically. Don't get confused with the terms "brute force", "neural net", etc. The main idea of the author is that AI that uses brute force, simpler statistical methods, NN, etc, wins over AI that tries to impleme…

Hmm, I'm not sure I see the difference. Why is it not "symbolic"? The symbols that construct the neural network are what encodes translation invariance -- not some vector of reals.

Symbolic as in "symbolic algebra systems", "symbolic AI", etc [1]. Not as in having some symbols in the code for a NN.

A NN doesn't work with the domain objects directly and abstractly (e.g. considering a face, facial features, smiles, etc as first class things and doing some kind of symbolic manipulation at that level).

It crunches numbers that encode patterns capturing those things, but its logic is all about numbers, links between one layer and another, and so on -- it's not a program dealing with high level abstract entities.

To put it in another way, it's the difference between teaching, say, Prolog to identify some concept and a NN to do the same.

E.g. from the link "The most successful form of symbolic AI is expert systems, which use a network of production rules. Production rules connect symbols in a relationship similar to an If-Then statement. The expert system processes the rules to make deductions and to determine what additional information it needs, i.e. what questions to ask, using human-readable symbols."

A NN does nothing like that (not in any immediate, first class, way, where the rules are expressed as plain rules given by the programmer, like "foo is X", "bar has the Y property", etc).

Here's another way to see it: how you'd solve a linear equation with regular algebra (the steps and transformation etc), and how a NN would encode the same.

A symbolic algebra system will let you express an equation in symbolic form (more or less like a mathematician would write it), and even show you all the intermediate steps you'd take until the solution.

A NN trained to solve the same type of equations doesn't do that (and can't). It just tells you the answer (or an approximation thereof).

[1] https://en.wikipedia.org/wiki/Symbolic_artificial_intelligen...

Post reply on HN