Evolution Is the New Deep Learning • Hacker News Board

Re: Evolution Is the New Deep Learning

#81

Post by resu_nimda » Thu, Mar 22, 2018, 7:46 PM UTC

pas wrote: post #73

resu_nimda wrote: post #64

Earlier quoted context omitted.

1) One of the biggest reasons they fell out of favor for more "mathematical" approaches was that no one could really explain why exactly they worked. Kind of like how nobody can really explain how the brain works, or life in general. My gut feeling is that it is hubris to think that we are going to "figure out" intelligence with increasingly sophisticated mathematical models anytime soon. We are not giving proper cre…

> We are not giving proper credit to how complex it is, and the multi-billion year developmental process that it took. Or we are simply not ready to accept that it's simply a big book of heuristics fine-tuned over biological eons. It's just big. We have too many interwoven, interdependent, synergistic faculties. Input, output, and a lot of mental stuff for making the right connections between the ins and the outs. Th…

Your last paragraph seems to contain the kind of overconfidence that I'm talking about. I don't understand how you can say "consciousness is simply X" or "it's easy to do that [if you handwave away the hard parts]." Clearly it's not that simple or easy, or we would have done it.

We can't even create life from non-life. How can we begin to understand all the stuff you're talking about that's been layered on top? We don't understand this stuff well enough to just handwave it away as unimportant or trivial.

Re: Evolution Is the New Deep Learning

#82

Post by ScottBurson » Thu, Mar 22, 2018, 7:47 PM UTC

hcrisp wrote: post #58

PeterisP wrote: post #54

Earlier quoted context omitted.

DL models don't often get stuck at local optima. In theory, they could be vulnerable to that, but in practice they are not, it simply doesn't happen in most practical supervised learning applications. I'm not up to date on theoretical research about this topic, but as far as I recall there are some interesting demonstrations on realistic problems showing that all the different "local" optima resulting from different…

The theories I've heard for why you don't get stuck in local optima when using deep learning include: 1. It's hard to get stuck in multidimensional space 2. There are more saddles than convex local optima 3. There are many local optima, but they are all useful 4. Something related to spin glass theory (which I don't understand) 5. There is no theory, or we haven't found it yet; all we know is that it works in practic…

I think it's a combination of (1) and (2): in a high-dimensional space, for a local optimum to be convex it has to be convex in every dimension, the probability of which falls off exponentially in the number of dimensions. So in practice, they're all saddles.

Re: Evolution Is the New Deep Learning

#83

Post by padthai » Thu, Mar 22, 2018, 7:47 PM UTC

pas wrote: post #73

resu_nimda wrote: post #64

Earlier quoted context omitted.

1) One of the biggest reasons they fell out of favor for more "mathematical" approaches was that no one could really explain why exactly they worked. Kind of like how nobody can really explain how the brain works, or life in general. My gut feeling is that it is hubris to think that we are going to "figure out" intelligence with increasingly sophisticated mathematical models anytime soon. We are not giving proper cre…

> We are not giving proper credit to how complex it is, and the multi-billion year developmental process that it took. Or we are simply not ready to accept that it's simply a big book of heuristics fine-tuned over biological eons. It's just big. We have too many interwoven, interdependent, synergistic faculties. Input, output, and a lot of mental stuff for making the right connections between the ins and the outs. Th…

There is plenty of magic going on. Today we cannot replicate or understand how emergent properties born of biological structures. Not even in "simple" systems as the metabolic pathways.

Re: Evolution Is the New Deep Learning

#84

Post by [deleted] » Thu, Mar 22, 2018, 7:49 PM UTC

pishpash wrote: post #75

Would the B-school types leave science and engineering to real scientists and engineers, please?

[deleted]

Re: Evolution Is the New Deep Learning

#85

Post by azinman2 » Thu, Mar 22, 2018, 7:50 PM UTC

levesque wrote: post #32

azinman2 wrote: post #22

Everything old is new again

I guess we should get ready for another AI/neural network winter. Kernel methods will be popular again!

Only if you can fit the buzzwords neuro or deep into empirical/statistical methods ;)

Re: Evolution Is the New Deep Learning

#86

Post by _mhr_ » Thu, Mar 22, 2018, 7:52 PM UTC

yters wrote: post #59

The only thing EAs have going for them is a biological metaphor, the magic of Darwinian evolution, fountain of endless novelty. But, modern science shows evolution does not really work in a Darwinian manner, so thus the metaphor ends.

Can you provide a source for what modern science you're referring to? What alternative to Darwinian evolution do you have in mind?

Re: Evolution Is the New Deep Learning

#87

Post by [deleted] » Thu, Mar 22, 2018, 7:58 PM UTC

padthai wrote: post #83

pas wrote: post #73

Earlier quoted context omitted.

> We are not giving proper credit to how complex it is, and the multi-billion year developmental process that it took. Or we are simply not ready to accept that it's simply a big book of heuristics fine-tuned over biological eons. It's just big. We have too many interwoven, interdependent, synergistic faculties. Input, output, and a lot of mental stuff for making the right connections between the ins and the outs. Th…

There is plenty of magic going on. Today we cannot replicate or understand how emergent properties born of biological structures. Not even in "simple" systems as the metabolic pathways.

[deleted]

Re: Evolution Is the New Deep Learning

#88

Post by antirez » Thu, Mar 22, 2018, 7:58 PM UTC

andreyk wrote: post #77

antirez wrote: post #76

Earlier quoted context omitted.

Genetic algorithms yes but genetic programming is actually able to invent new things. Possibly optimizing the set of weight of a neural network to solve a given problem is more similar to genetic programming, if you see the NN as a computational unit.

Genetic algorithms 'invent new things' by doing exactly the same thing as other optimization methods - tweaking the values of particular parameters. The only difference in is how they do so.

Optimizing genes leading to an intelligent neural network is what biology did basically. If your optimization problem changes the behavior of something able to compute in order to maximize the ability to solve a task, then that's inventing things IMHO.

Re: Evolution Is the New Deep Learning

#89

Post by pas » Thu, Mar 22, 2018, 7:59 PM UTC

padthai wrote: post #83

pas wrote: post #73

Earlier quoted context omitted.

> We are not giving proper credit to how complex it is, and the multi-billion year developmental process that it took. Or we are simply not ready to accept that it's simply a big book of heuristics fine-tuned over biological eons. It's just big. We have too many interwoven, interdependent, synergistic faculties. Input, output, and a lot of mental stuff for making the right connections between the ins and the outs. Th…

There is plenty of magic going on. Today we cannot replicate or understand how emergent properties born of biological structures. Not even in "simple" systems as the metabolic pathways.

We mapped the whole genome and connectome of C. elegans, no? And most of that is understood. For example, it seems to be a good model for substance addiction (especially for nicotine). That seems a pretty complex emergent behavior to me.

If you mean bigger biology, yes, sure, we don't have a full map of functional genomics for humans, but we're getting there.

Or maybe not, maybe it's so exponentially more complex, that it'd take as much time to understand it as it took for evolution to work it out. (Especially considering that evolution played with every individual, whereas we like to constrain our data gathering to non-aggressive methods.)

Re: Evolution Is the New Deep Learning

#90

Post by avip » Thu, Mar 22, 2018, 8:02 PM UTC

evolutioner wrote: post #72

mav3r1ck wrote: post #15

Having studied this extensively back when they were called Genetic Algorithms, I would like to offer a few insights. 1) One of the biggest reasons they fell out of favor for more "mathematical" approaches was that no one could really explain why exactly they worked. It makes sense on the surface that "survival of the fittest" and doing something akin to multiple stochastic gradient descents would work, but no one has…

Sentient employee here. I'll give an example on a problem for which we use evolutionary algorithms: website optimization. Say you want to try many various changes like the title of your page, the color of the background, the position of your buy button etc. We solve this problem by trying out random variations of these websites - like A/B testing with more candidates - and by crossing the best performing ones to crea…

Ok... so... let's say your A/B only optimizes the colour and location of a single CTA button, and conversion rate is your fitness. 2 previous generations have top/blue and bottom/red. How does the next one look?