Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Evolution Is the New Deep Learning

sentient.ai

Post reply on HN
Print view View original

21–30 of 242 posts

Re: Evolution Is the New Deep Learning

#21

Post by chrisfosterelli »

These "evolved neurons" look ridiculously more complicated and expensive to evaluate than a base neuron for a minor improvement in performance. The Music Maker paper mentioned that LSTMs haven't changed in "25 years", but didn't mention the 2014 Gated Recurrent Units that have become so popular specifically because they are much more simple to understand and evaluate. This approach seems to be going in the opposite direction.

I guess the idea of them being evolved is that you don't _need_ to understand them, but evaluation performance is definitely a concern and doesn't seem to be addressed much. Interesting early work though!

Re: Evolution Is the New Deep Learning

#23

Post by bitL »

mav3r1ck wrote: post #15

Having studied this extensively back when they were called Genetic Algorithms, I would like to offer a few insights. 1) One of the biggest reasons they fell out of favor for more "mathematical" approaches was that no one could really explain why exactly they worked. It makes sense on the surface that "survival of the fittest" and doing something akin to multiple stochastic gradient descents would work, but no one has…

Crossing two genotypes and making mutations so that they produce consistent encoding was for me what put genetic algorithms to purely academic area - for things like TSP I either couldn't come up with good crossing function or it was producing mostly the same results after applying some "consistentification".

Re: Evolution Is the New Deep Learning

#24

Post by fnbr »

I think this is true, broadly. Biological evolution has a large advantage in that it introduces a large amount of exogenous variation into the system (it just randomly changes DNA via a number of ways, and selects the "best" changes). This can be good as it allows you to do a lot more exploration than with a comparable gradient based approach, as the gradient based approach can't explore entirely new parameters; it can only look at parameters that have already been shown to be good. This is why, for instance, dead neurons are a problem with ReLUs. I don't think (but have not proof) that this would be a problem with evolution.

I think that evolutionary approaches are a lot better at the exploration phase, but weak at exploitation; gradient methods tend to be the opposite.

I think some sort of combination of the two, like Max Jaderberg's "Population based training of neural networks" [1], is the way to go.

[1]: https://deepmind.com/blog/population-based-training-neural-n...

Re: Evolution Is the New Deep Learning

#25

Post by mav3r1ck »

ewjordan wrote: post #19
mav3r1ck wrote: post #15

Having studied this extensively back when they were called Genetic Algorithms, I would like to offer a few insights. 1) One of the biggest reasons they fell out of favor for more "mathematical" approaches was that no one could really explain why exactly they worked. It makes sense on the surface that "survival of the fittest" and doing something akin to multiple stochastic gradient descents would work, but no one has…

Wait, I'm confused...you're saying genetic approaches fell out of favor because they're basically just stochastic gradient descent? Most of modern DL relies heavily on SGD at various points during training. My impression is that they fell out of favor precisely because don't actually use any gradients, and end up converging on good maxima slower than you could if you used the gradients from the net. Am I off base her…

You are not off base at all, thanks for clarify and sorry for the confusion, I did not mean to say it was using gradient descent. It's been a while. The term I was thinking of was multiple "simulated annealing".

Re: Evolution Is the New Deep Learning

#26

Post by koube »

mav3r1ck wrote: post #15

Having studied this extensively back when they were called Genetic Algorithms, I would like to offer a few insights. 1) One of the biggest reasons they fell out of favor for more "mathematical" approaches was that no one could really explain why exactly they worked. It makes sense on the surface that "survival of the fittest" and doing something akin to multiple stochastic gradient descents would work, but no one has…

Where's the posts on explainable AI? I did a ctrl+f but didn't see any.

Re: Evolution Is the New Deep Learning

#29

Post by yorwba »

ewjordan wrote: post #19
mav3r1ck wrote: post #15

Having studied this extensively back when they were called Genetic Algorithms, I would like to offer a few insights. 1) One of the biggest reasons they fell out of favor for more "mathematical" approaches was that no one could really explain why exactly they worked. It makes sense on the surface that "survival of the fittest" and doing something akin to multiple stochastic gradient descents would work, but no one has…

Wait, I'm confused...you're saying genetic approaches fell out of favor because they're basically just stochastic gradient descent? Most of modern DL relies heavily on SGD at various points during training. My impression is that they fell out of favor precisely because don't actually use any gradients, and end up converging on good maxima slower than you could if you used the gradients from the net. Am I off base her…

If you mutate genomes by small additive modifications to a vector of continuous parameters, then taking lots of samples and keeping the best is essentially a stochastic approximation to gradient descent. However, unlike the SGD used in deep learning, it doesn't make use of calculus and therefore requires many more samples (exponentially more, in the worst case) to get a gradient of equivalent accuracy. I.e. it's slow.

If your mutations aren't small, or your parameters are not continuously valued, or your fitness function is hard to differentiate analytically, genetic algorithms might still come out ahead.

Re: Evolution Is the New Deep Learning

#30

Post by Geee »

mav3r1ck wrote: post #15

Having studied this extensively back when they were called Genetic Algorithms, I would like to offer a few insights. 1) One of the biggest reasons they fell out of favor for more "mathematical" approaches was that no one could really explain why exactly they worked. It makes sense on the surface that "survival of the fittest" and doing something akin to multiple stochastic gradient descents would work, but no one has…

Isn't that the same question as to why the whole universe exists? Why chaos results in order? Isn't there a proof that 'given such random process, an universe like we have is inevitably created'?
Post reply on HN