Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Evolution Is the New Deep Learning

sentient.ai

Post reply on HN
Print view View original

171–180 of 242 posts

Re: Evolution Is the New Deep Learning

#171

Post by nickpsecurity »

antirez wrote: post #76
andreyk wrote: post #71

"Like Deep Learning (DL), EC was introduced decades ago, and it is currently experiencing a similar boost from the available big compute and big data. However, it addresses a distinctly different need: Whereas DL focuses on modeling what we already know, EC focuses on creating new knowledge." What utter nonsense. Genetic Algorithms do exactly the same thing that Deep Learning methods do: optimize a function for a par…

Genetic algorithms yes but genetic programming is actually able to invent new things. Possibly optimizing the set of weight of a neural network to solve a given problem is more similar to genetic programming, if you see the NN as a computational unit.

To support your claim, here's plenty of examples of GP inventing new things:

http://www.genetic-programming.org/combined.php

Here's a fun article from Popular Science on Koza and his "invention machine" that readers might enjoy:

https://www.popsci.com/scitech/article/2006-04/john-koza-has...

Re: Evolution Is the New Deep Learning

#172

Post by oldandtired »

TaylorAlexander wrote: post #99
resu_nimda wrote: post #64

Earlier quoted context omitted.

1) One of the biggest reasons they fell out of favor for more "mathematical" approaches was that no one could really explain why exactly they worked. Kind of like how nobody can really explain how the brain works, or life in general. My gut feeling is that it is hubris to think that we are going to "figure out" intelligence with increasingly sophisticated mathematical models anytime soon. We are not giving proper cre…

While I can’t predict that we can solve intelligence completely any time soon, I would argue that machine intelligence allows us to examine the phenomena of intelligence more thoroughly than anything else in the past. I would expect this to yield new insights. So at a minimum, I’d think we can learn more now than we have been able to before. Maybe that will lead to a great increase in understanding of intelligence or…

The only insights will be related to what insights we get about the programming we are doing with machines. It is an entirely different order to providing insights into even the simplest of organic neurological systems.

Computer simulations can give insight into simple systems like water flows, etc. Simulating more complex systems like a single living cells or any system built on livings cells would require systems that would not really be worth building. It would be simpler to use cells directly.

Re: Evolution Is the New Deep Learning

#173

Post by inverse_pi »

Houshalter wrote: post #151
inverse_pi wrote: post #103

Earlier quoted context omitted.

My thesis was on Generic Algorithm. I stopped and started working on Deep Learning mainly because like you said, GAs don't really have a strong mathematical foundation. Ironically, no one could really explain why CNNs work mathematically either. I've heard a lot of hand-wavy arguments about local search, local sensitivity, etc. However, no one could really prove anything meaningful. There are some papers around certa…

Why should we expect there to be any mathematical foundation to this stuff? It's quite possible to imagine an alternate universe where GAs and neural nets don't work. Because they have different datasets that don't fit the structure of NNs well. Or problems that happen to not be solvable by the search strategies of GAs. In fact we have many such problems in our own universe. I can give many examples of things NNs and…

why does the existence of such problems disprove the existence of a mathematical foundation? A well-founded mathematical foundation would prove/predict/explain why such problems don't "fit" with the "structure of NNs" with precise lower/upper bounds. Anything that works, and especially everything that doesn't work, must have an explanation. God doesn't play dice.

Re: Evolution Is the New Deep Learning

#174

Post by goatlover »

dorgo wrote: post #120
resu_nimda wrote: post #64

Earlier quoted context omitted.

1) One of the biggest reasons they fell out of favor for more "mathematical" approaches was that no one could really explain why exactly they worked. Kind of like how nobody can really explain how the brain works, or life in general. My gut feeling is that it is hubris to think that we are going to "figure out" intelligence with increasingly sophisticated mathematical models anytime soon. We are not giving proper cre…

> My gut feeling is that it is hubris to think that we are going to "figure out" intelligence with increasingly sophisticated mathematical models anytime soon. We did it already. Compter understand language, translate it, react to it. They can recognize items on a picture. Is there a task left which can't be done by computers better and faster than by humans? >Almost by definition, if we can analytically understand i…

> Is there a task left which can't be done by computers better and faster than by humans?

All the tasks humans still earn money doing. And given that we're nowhere near full automation, I'd say it's quite a few tasks.

Re: Evolution Is the New Deep Learning

#176

Post by goatlover »

TheOtherHobbes wrote: post #36
goatlover wrote: post #13

Earlier quoted context omitted.

350,000 species of beetles.

In music, evolution produced Bach, so it can't really be considered a failure. It took a while though, and it's possible alternative approaches would have been produced a similar result more efficiently.

It also produced Justin Bieber and Weird Al. Point being, if the goal of your evolutionary algorithm is Bach, how do you filter for that result among the other million possibilities? And what makes Bach the success of music? Maybe more people prefer listening to Taylor Swift.

Re: Evolution Is the New Deep Learning

#177

Post by JunkDNA »

resu_nimda wrote: post #64
mav3r1ck wrote: post #15

Having studied this extensively back when they were called Genetic Algorithms, I would like to offer a few insights. 1) One of the biggest reasons they fell out of favor for more "mathematical" approaches was that no one could really explain why exactly they worked. It makes sense on the surface that "survival of the fittest" and doing something akin to multiple stochastic gradient descents would work, but no one has…

1) One of the biggest reasons they fell out of favor for more "mathematical" approaches was that no one could really explain why exactly they worked. Kind of like how nobody can really explain how the brain works, or life in general. My gut feeling is that it is hubris to think that we are going to "figure out" intelligence with increasingly sophisticated mathematical models anytime soon. We are not giving proper cre…

Taking your point a step further, I often wonder if mathematics is a local minimum for humans. It has been so damn effective in so many ways that we can’t imagine that there might be some other mechanism for solving hard problems. In cases where the mathematics gets really complex, I wonder if this is a hint that there’s some other way to represent the situation.

Re: Evolution Is the New Deep Learning

#178

Post by breuleux »

Houshalter wrote: post #145
breuleux wrote: post #126

Earlier quoted context omitted.

On the other hand, even if evolutionary algorithms require a lot of samples, they are embarrassingly parallel: you can easily try all samples simultaneously. If you have enough resources to throw at the problem, it can be faster (although more resource-intensive) to estimate the gradient this way than to compute an accurate gradient analytically.

SGD is embarrassingly parallel as well. You can train a net on several different examples simultaneously and combine the gradients or learned weights. The reason it's not done so much is because the bandwidth of moving huge numbers of gradients or weights between computers is pretty significant. There's been all sorts of research into compressing them or reducing the precision. However this is a problem for evolution…

Not exactly. The required bandwidth for the evolutionary strategy is actually very small: if every node knows each other's random seed, they can reconstruct the best model themselves, using the seed of whichever node declares the best result. There is no need to transfer any weights. Of course, this trick only works if it's cheap to compute the weights other nodes are using, which is not the case for SGD. So evolutionary strategies have an advantage here, even if it's not a decisive one.

Re: Evolution Is the New Deep Learning

#179

Post by andreyk »

dmix wrote: post #147
IanCal wrote: post #96

Earlier quoted context omitted.

It's banal but sensible in response to someone claiming a large difference between two fundamentally similar things. Either both or neither "invents".

As a non-expert in the field my impression from reading the article wasn't that they were saying there was a large difference, or it's a whole new approach, only that is was a progression over DL. The entire article repeatedly relates everything in the context of what DL accomplishes and only introduces EC as an better optimized version. Which doesn't lead one to think they are significantly different approaches but…

The article makes this claim: "it addresses a distinctly different need: Whereas DL focuses on modeling what we already know, EC focuses on creating new knowledge."

This is just wrong.

Re: Evolution Is the New Deep Learning

#180

Post by Cacti »

sampo wrote: post #105
Cacti wrote: post #69

Earlier quoted context omitted.

Yep. The original wave of genetic algorithms largely depended on some hand-wavy "building block" ideas that no one could really prove. It turned out that it was because proving them is impossible in the general sense, as we found out from the NFL theorems in the mid-to-late 90s, and it wasn't even clear the field had a scientific basis at all. So I was surprised to see them make a return about a decade later. Hopeful…

> as we found out from the NFL theorems in the mid-to-late 90s, and it wasn't even clear the field had a scientific basis at all. NFL theorems are, should I say, purely theoretical and provide no insight on real-world problems. Say we try to find a function that is an optimal solution to something. NFL theorems consider the space of all possible functions, the overwhelming majority of which are discontinuous. Whereas…

I hear this sentiment a lot lately, and I do not agree with it at all, you're basically just hand-waving the issue away and retroactively deciding what is a real world, practical problem or not after we've already found a solution to those problems. The NFL theorems are not inherently worthless, you are making them worthless because you are throwing away nearly all of the possible problem space, under the assumption that space is not relevant, but that is just a guess, you don't actually know.

I mean, yeah, sure, not knowing if we've hit the global minimum on some optimization problem may not matter as humans, because no one gives a damn, but that is a completely arbitrary line, not a mathematical one. Too often I hear people say "well, NFL is irrelevant" as an argument for why they are mathematically correct, or why their results are mathematically significant, and that's simply a load of crap. Maybe you're right, maybe you're wrong, but you're basically just throwing darts at a dartboard.

Post reply on HN