Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Evolution Is the New Deep Learning

sentient.ai

Post reply on HN
Print view View original

201–210 of 242 posts

Re: Evolution Is the New Deep Learning

#201

Post by jmmcd »

andreyk wrote: post #71

"Like Deep Learning (DL), EC was introduced decades ago, and it is currently experiencing a similar boost from the available big compute and big data. However, it addresses a distinctly different need: Whereas DL focuses on modeling what we already know, EC focuses on creating new knowledge." What utter nonsense. Genetic Algorithms do exactly the same thing that Deep Learning methods do: optimize a function for a par…

> Genetic Algorithms do exactly the same thing that Deep Learning methods do: optimize a function for a particular criterion.

The difference is that for a GA, that can be any function, whereas for NNs it must be decomposable into a sum of partial objectives, and differentiable. Moreover, the result of training an NN is a function mapping a vector to a vector, whereas the result of training a GA could be an object drawn from any search space.

Re: Evolution Is the New Deep Learning

#202

Post by randomsearch »

jmmcd wrote: post #199
mav3r1ck wrote: post #15

Having studied this extensively back when they were called Genetic Algorithms, I would like to offer a few insights. 1) One of the biggest reasons they fell out of favor for more "mathematical" approaches was that no one could really explain why exactly they worked. It makes sense on the surface that "survival of the fittest" and doing something akin to multiple stochastic gradient descents would work, but no one has…

I think you've confused genetic algorithms with genetic programming. They're not the same thing. > I don't know how Genetic Algorithms/programming could be made 'explainable' as to why they achieved an optimal solution other than hand-waving to how evolution works in nature. This is quite confused. You're comparing different levels of the systems. In neural networks, we would like to know why a numerical model (which…

> (2) the reason the object is good is rather easy to see -- one just has to look at the objective function and verify that that the object has the desired properties;

Minor point: whether the result of evolution is easy to understand or not depends on the representation (encoding). Even GA results can be difficult to understand if they describe complex objects. In the case of GP, bloat (rapid increase in average program size in your population) can make results very difficult to interpret.

Re: Evolution Is the New Deep Learning

#203

Post by randomsearch »

ewjordan wrote: post #19
mav3r1ck wrote: post #15

Having studied this extensively back when they were called Genetic Algorithms, I would like to offer a few insights. 1) One of the biggest reasons they fell out of favor for more "mathematical" approaches was that no one could really explain why exactly they worked. It makes sense on the surface that "survival of the fittest" and doing something akin to multiple stochastic gradient descents would work, but no one has…

Wait, I'm confused...you're saying genetic approaches fell out of favor because they're basically just stochastic gradient descent? Most of modern DL relies heavily on SGD at various points during training. My impression is that they fell out of favor precisely because don't actually use any gradients, and end up converging on good maxima slower than you could if you used the gradients from the net. Am I off base her…

GAs are not "basically just stochastic gradient descent." That's the confusion.

Why did they fall out of favour? Fashion, perhaps a natural break in progress - hit a wall and couldn't get further. But there is plenty of GA research going on.

Re: Evolution Is the New Deep Learning

#204

Post by pas »

resource0x wrote: post #162
pas wrote: post #73

Earlier quoted context omitted.

> We are not giving proper credit to how complex it is, and the multi-billion year developmental process that it took. Or we are simply not ready to accept that it's simply a big book of heuristics fine-tuned over biological eons. It's just big. We have too many interwoven, interdependent, synergistic faculties. Input, output, and a lot of mental stuff for making the right connections between the ins and the outs. Th…

200 years ago there was nothing to understand in electricity: it was just a liquid :) https://en.wikipedia.org/wiki/Fluid_theory_of_electricity

There were experiments that were not explained by the liquid theory.

Now we have data and people for some reason want to claim that a theory with magical super complex and not-even-yet-describable and very-very-irreducible element(s) is a better fit than a good old box full of tiny yet specialized parts fine-tuned to work together over millions of years.

Re: Evolution Is the New Deep Learning

#205

Post by randomsearch »

jmmcd wrote: post #201
andreyk wrote: post #71

"Like Deep Learning (DL), EC was introduced decades ago, and it is currently experiencing a similar boost from the available big compute and big data. However, it addresses a distinctly different need: Whereas DL focuses on modeling what we already know, EC focuses on creating new knowledge." What utter nonsense. Genetic Algorithms do exactly the same thing that Deep Learning methods do: optimize a function for a par…

> Genetic Algorithms do exactly the same thing that Deep Learning methods do: optimize a function for a particular criterion. The difference is that for a GA, that can be any function, whereas for NNs it must be decomposable into a sum of partial objectives, and differentiable. Moreover, the result of training an NN is a function mapping a vector to a vector , whereas the result of training a GA could be an object dr…

I think there's a further difference - with some GAs (including GP, widely used by Sentient) the structure of the solution is "designed" alongside the parameterisation of that structure.

Most DL work parameter tunes a structure that is hacked by human guesswork.

The two approaches have strengths and weaknesses, and in no way does this make EC in general superior to DL, but perhaps different.

Re: Evolution Is the New Deep Learning

#206

Post by pas »

mrfusion wrote: post #165
pas wrote: post #73

Earlier quoted context omitted.

> We are not giving proper credit to how complex it is, and the multi-billion year developmental process that it took. Or we are simply not ready to accept that it's simply a big book of heuristics fine-tuned over biological eons. It's just big. We have too many interwoven, interdependent, synergistic faculties. Input, output, and a lot of mental stuff for making the right connections between the ins and the outs. Th…

Keep in mind the genome really isn’t big enough to store enough hueristics to make a functioning human.

Yes, of course, the problem of bootstrapping consciousness from blueprints of a human mind is that we depend on our parents' whole epigenetic and other extra informational make up, plus their support for years while our mind finishes setting up.

Re: Evolution Is the New Deep Learning

#208

Post by pas »

padthai wrote: post #163
ozy wrote: post #117

Earlier quoted context omitted.

Same way waves emerge in water. The rules that govern a system can create patterns, which themselves behave according to rules, but with a set of rules that was "hard to predict" from the underlying system.

That is exactly my point. We can use fluid dynamics and PDEs in waves. We understand some properties and processes. We are nowhere as close in biological system. I put the example of the metabolic pathways because last time checked (~2015) the most advanced things in the field were extremely simple and without any predictive power. Things like calculating the kernel of a stoichiometric matrix or the centrality of a n…

There's probably no need to go that low-level for modeling a mind, but of course the aggregate effects of biochemistry has to be taken into account (and it's full of non-linearities).

None of that means we don't understand the principles. I'd say it's pretty much like fusion. Yes, we know how the Sun works, but putting it into a bottle is a bit of a pickle, similarly with brains. (Except brains have a lot more complexity.)

Re: Evolution Is the New Deep Learning

#209

Post by cabalamat »

mav3r1ck wrote: post #15

Having studied this extensively back when they were called Genetic Algorithms, I would like to offer a few insights. 1) One of the biggest reasons they fell out of favor for more "mathematical" approaches was that no one could really explain why exactly they worked. It makes sense on the surface that "survival of the fittest" and doing something akin to multiple stochastic gradient descents would work, but no one has…

> I don't know how Genetic Algorithms/programming could be made 'explainable' as to why they achieved an optimal solution other than hand-waving to how evolution works in nature.

With either GA or ANN, you don't know that the algorithm has achieved the optimal solution and I would be surprised if you could ever prove that in the general case.

What you can know is that your algorithm performs well, e.g. by testing how well it plays go or drives a car or recognises faces, or whatever.

Re: Evolution Is the New Deep Learning

#210

Post by jmmcd »

randomsearch wrote: post #205
jmmcd wrote: post #201

Earlier quoted context omitted.

> Genetic Algorithms do exactly the same thing that Deep Learning methods do: optimize a function for a particular criterion. The difference is that for a GA, that can be any function, whereas for NNs it must be decomposable into a sum of partial objectives, and differentiable. Moreover, the result of training an NN is a function mapping a vector to a vector , whereas the result of training a GA could be an object dr…

I think there's a further difference - with some GAs (including GP, widely used by Sentient) the structure of the solution is "designed" alongside the parameterisation of that structure. Most DL work parameter tunes a structure that is hacked by human guesswork. The two approaches have strengths and weaknesses, and in no way does this make EC in general superior to DL, but perhaps different.

I agree, but I intended that point as part of my final point -- an EA can draw a solution from any search space, including a space of trees, for example, as in GP.
Post reply on HN