Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Evolution Is the New Deep Learning

sentient.ai

Post reply on HN
Print view View original

181–190 of 242 posts

Re: Evolution Is the New Deep Learning

#181

Post by jamesrcole »

resu_nimda wrote: post #64
mav3r1ck wrote: post #15

Having studied this extensively back when they were called Genetic Algorithms, I would like to offer a few insights. 1) One of the biggest reasons they fell out of favor for more "mathematical" approaches was that no one could really explain why exactly they worked. It makes sense on the surface that "survival of the fittest" and doing something akin to multiple stochastic gradient descents would work, but no one has…

1) One of the biggest reasons they fell out of favor for more "mathematical" approaches was that no one could really explain why exactly they worked. Kind of like how nobody can really explain how the brain works, or life in general. My gut feeling is that it is hubris to think that we are going to "figure out" intelligence with increasingly sophisticated mathematical models anytime soon. We are not giving proper cre…

> going off of Arthur C Clarke's definition, life is magic. And we're trying to create something equally magical.

I assume you're referring to his "any sufficiently advanced technology is indistinguishable from magic"? If so, you're misrepresenting it, because he's clearly saying it's not magic, it just appears that way to the unadvanced. And there's a big difference between "appears to be" and "is".

Re: Evolution Is the New Deep Learning

#182

Post by theparanoid »

JunkDNA wrote: post #177
resu_nimda wrote: post #64

Earlier quoted context omitted.

1) One of the biggest reasons they fell out of favor for more "mathematical" approaches was that no one could really explain why exactly they worked. Kind of like how nobody can really explain how the brain works, or life in general. My gut feeling is that it is hubris to think that we are going to "figure out" intelligence with increasingly sophisticated mathematical models anytime soon. We are not giving proper cre…

Taking your point a step further, I often wonder if mathematics is a local minimum for humans. It has been so damn effective in so many ways that we can’t imagine that there might be some other mechanism for solving hard problems. In cases where the mathematics gets really complex, I wonder if this is a hint that there’s some other way to represent the situation.

Computational processes have picked up where mathematics stops.

Re: Evolution Is the New Deep Learning

#183

Post by VVyattPrentice »

pas wrote: post #95
resu_nimda wrote: post #81

Earlier quoted context omitted.

Your last paragraph seems to contain the kind of overconfidence that I'm talking about. I don't understand how you can say "consciousness is simply X" or "it's easy to do that [if you handwave away the hard parts]." Clearly it's not that simple or easy, or we would have done it. We can't even create life from non-life. How can we begin to understand all the stuff you're talking about that's been layered on top? We do…

I'm assuming a simpler model, no need for magic, because so far I don't see what behavior/data this simple model cannot explain. > Clearly it's not that simple or easy, or we would have done it. We don't have the computational power yet. Not to mention the vast amount of development required. Think of the climate models, that are huge (millions of lines of code), but they're still nowhere near complete enough, and th…

If intelligence/sentience isn't magic for you, then consider existence. How strange it is to be anything at all.

Re: Evolution Is the New Deep Learning

#184

Post by hnuser1234 »

resu_nimda wrote: post #64
mav3r1ck wrote: post #15

Having studied this extensively back when they were called Genetic Algorithms, I would like to offer a few insights. 1) One of the biggest reasons they fell out of favor for more "mathematical" approaches was that no one could really explain why exactly they worked. It makes sense on the surface that "survival of the fittest" and doing something akin to multiple stochastic gradient descents would work, but no one has…

1) One of the biggest reasons they fell out of favor for more "mathematical" approaches was that no one could really explain why exactly they worked. Kind of like how nobody can really explain how the brain works, or life in general. My gut feeling is that it is hubris to think that we are going to "figure out" intelligence with increasingly sophisticated mathematical models anytime soon. We are not giving proper cre…

>The current industry approaches are great for extracting certain kinds of value out of large data sets, but in terms of producing a result that could even begin to be considered as interesting as life (i.e. AGI or "strong AI"), I believe we will have to rely on creating a system whose inner workings are too complex for us to understand.

Speak for yourself.

Re: Evolution Is the New Deep Learning

#185

Post by clicea »

jcora wrote: post #152
pas wrote: post #73

Earlier quoted context omitted.

> We are not giving proper credit to how complex it is, and the multi-billion year developmental process that it took. Or we are simply not ready to accept that it's simply a big book of heuristics fine-tuned over biological eons. It's just big. We have too many interwoven, interdependent, synergistic faculties. Input, output, and a lot of mental stuff for making the right connections between the ins and the outs. Th…

I'd call this pulling a Dennett: trivializing complexity to something that cannot or just doesn't have to be explained. Being unable to conceive consciousness at this moment doesn't mean there's nothing to conceive of: even if we never get to the final satisfactory answer, there is undoubtedly much more room left for useful concepts we don't have yet, around or inside this idea

I'd call bullshit. Dennett's argument not that the brain is complex but that it's not obvious it is not reductible.

Re: Evolution Is the New Deep Learning

#186

Post by Houshalter »

inverse_pi wrote: post #173
Houshalter wrote: post #151

Earlier quoted context omitted.

Why should we expect there to be any mathematical foundation to this stuff? It's quite possible to imagine an alternate universe where GAs and neural nets don't work. Because they have different datasets that don't fit the structure of NNs well. Or problems that happen to not be solvable by the search strategies of GAs. In fact we have many such problems in our own universe. I can give many examples of things NNs and…

why does the existence of such problems disprove the existence of a mathematical foundation? A well-founded mathematical foundation would prove/predict/explain why such problems don't "fit" with the "structure of NNs" with precise lower/upper bounds. Anything that works, and especially everything that doesn't work, must have an explanation. God doesn't play dice.

Maybe there's a reason. But I don't think you will ever be able to "prove" it. In any kind of formal way that would be satisfying to mathematicians.

Solomonoff induction is the best attempt to try to formalize machine learning. And in theory any machine learning algorithm that works, works because it approximates Solomonoff induction somehow. But proving anything approximates Solomonoff induction is absurdly difficult or impossible. Because it's incomputable and involves the search space of all possible computer programs.

Re: Evolution Is the New Deep Learning

#187

Post by Houshalter »

breuleux wrote: post #178
Houshalter wrote: post #145

Earlier quoted context omitted.

SGD is embarrassingly parallel as well. You can train a net on several different examples simultaneously and combine the gradients or learned weights. The reason it's not done so much is because the bandwidth of moving huge numbers of gradients or weights between computers is pretty significant. There's been all sorts of research into compressing them or reducing the precision. However this is a problem for evolution…

Not exactly. The required bandwidth for the evolutionary strategy is actually very small: if every node knows each other's random seed, they can reconstruct the best model themselves, using the seed of whichever node declares the best result. There is no need to transfer any weights. Of course, this trick only works if it's cheap to compute the weights other nodes are using, which is not the case for SGD. So evolutio…

Well that would work for simple hillclimbing. In a full genetic algorithm, every member of the population is the product of thousands of previous individuals.

Re: Evolution Is the New Deep Learning

#188

Post by _mhr_ »

yters wrote: post #123
_mhr_ wrote: post #86

Earlier quoted context omitted.

Can you provide a source for what modern science you're referring to? What alternative to Darwinian evolution do you have in mind?

https://en.wikipedia.org/wiki/Extended_evolutionary_synthesi... Essentially, variation comes from many other sources than random mutation on DNA. Random mutation itself is seen as mostly a bad source of variation, leading to destruction of the genome.

That's a very interesting hypothesis, thank you for the link. However, it doesn't seem to support what you're claiming about mutations, from my cursory reading. Can you provide a source for this claim that random mutation is a bad source of variation?

http://extendedevolutionarysynthesis.com/about-the-ees/why-i... explicitly rejects the idea of a revolution regarding mutation and other genetic sources of variation:

> How can the EES seek 'profound change' and yet disavow 'revolution'?

> All recognized causes of both evolution (e.g. natural selection, genetic drift, mutation, etc) and inheritance (e.g. genes), as well as the vast body of empirical and theoretical findings generated by the field of evolutionary biology, are accepted by the EES.

> Hence the EES does not entail a rejection of current understanding within the field, and does not require revolution. The EES seeks only to supplement the existing causal framework through recognition of additional causes of evolution (e.g. developmental bias) and inheritance (e.g. epigenetic inheritance), entirely complementary to those long-established within the field. Nevertheless, should these additional processes prove to be important, the conceptual change to evolutionary biology could very well be fundamental.

Although not in your source, I did find the following. Known as Drake's rule: as a genome's size increases, the mutation rate also tends to decrease. https://en.wikipedia.org/wiki/Genome_size#Drake's_rule. A recent update to Drake's rule suggests that mutation rate is also negatively correlated with population size too, not just genome size (https://www.ncbi.nlm.nih.gov/pmc/articles/PMC3494944/). And another interesting article about mutation in reproduction: https://www.nature.com/articles/ncomms15183 (somatic mutation is two orders of magnitude higher than germline mutation). This is intriguing, though I have to wonder how the genome ever grows if mutation supposedly becomes so strict over time as to become a non-factor (transposons, gene duplication?). It's worth noting that mutations do occur more frequently in non-coding sections of the genome, which may not be accounted for in the studies I've linked to.

Still, regardless of my sources, I would like to see you provide support from specifically EES proponents for the devaluation of mutation as a source of genetic variation, as you implied.

Re: Evolution Is the New Deep Learning

#189

Post by textor »

pas wrote: post #95
resu_nimda wrote: post #81

Earlier quoted context omitted.

Your last paragraph seems to contain the kind of overconfidence that I'm talking about. I don't understand how you can say "consciousness is simply X" or "it's easy to do that [if you handwave away the hard parts]." Clearly it's not that simple or easy, or we would have done it. We can't even create life from non-life. How can we begin to understand all the stuff you're talking about that's been layered on top? We do…

I'm assuming a simpler model, no need for magic, because so far I don't see what behavior/data this simple model cannot explain. > Clearly it's not that simple or easy, or we would have done it. We don't have the computational power yet. Not to mention the vast amount of development required. Think of the climate models, that are huge (millions of lines of code), but they're still nowhere near complete enough, and th…

> We don't have the computational power yet

Certainly you do realise that this has been a moving goalpost for half a century? It seems that lately people started to avoid giving a concrete estimate of the power required, though. It was so easy in the 90-s! «Human visual/verbal system processes gygabytes per second/has n flops to the xth» or the like. Well, now we have that and more; how come a deeper modeling, a finer processing, a more complicated network has come to be needed?

And your examples are incorrect, for example Navier-Stokes equations plus some general physics knowledge always have allowed us to estimate how much data we need for a certain fidelity of a finite-term weather forecast. Certainly we need more for a complete climate model, but we know what we need. No such thing about the brain.

It's an easy way to score some rationality points by voicing rejection of "magic", but it's a strawman. Nobody will bother arguing for a mythical homunculus in the seat of the soul, nor even for a concise formula summing up the workings of the mind. Pick harder targets. "It's just big" or "it's just a bunch of heuristics cobbled together" is a non-explanation. The brain is not a Rube Goldberg machine that manages to produce any sort of work simply due to its excessive complexity – it is energetically economical, taking into account that neurons are living cells that need to sustain their metabolism and not merely "compute" when provided with energy. Its discrete elements aren't really small by today's standards, nor are they fast. The number of synapses is ridiculous, but since they aren't independent, at a glance it doesn't add that much complexity too (unless we abandon reason and emulate everything close to the physical level).

Yet we have failed to realistically emulate a worm. By all accounts we have enough power for 302 neurons already. There's no workload to give to overwhelm available supercomputers. It's knowledge and understanding that we lack, and it's high time to give up on the delusion that more power, naturally coming in the future, will somehow enable a creation of predictive brain model, for this would truly be magic.

Re: Evolution Is the New Deep Learning

#190

Post by dorgo »

hueving wrote: post #164
dorgo wrote: post #120

Earlier quoted context omitted.

> My gut feeling is that it is hubris to think that we are going to "figure out" intelligence with increasingly sophisticated mathematical models anytime soon. We did it already. Compter understand language, translate it, react to it. They can recognize items on a picture. Is there a task left which can't be done by computers better and faster than by humans? >Almost by definition, if we can analytically understand i…

Yep, we're definitely done: http://rocknrollnerd.github.io/ml/2015/05/27/leopard-sofa.ht...

Show this leopard sofa to 100 humans for 1 second each and see how many will make the same mistake. We are 99% there and you point to the 1% to prove how bad we failed. Sure, the error rate will go down even further in the next years. But what we have is more than good enough to be used in commercial products. It's not like we are trying to win a contest man vs. machine (which we will or already have).
Post reply on HN