21–30 of 47 posts
Note that using genetic algorithms to come up with initial weights for neural networks was the state of the art in the late 90s/early 2000s. So this paper is not as novel as it seems, but it's good to have it for reference. My feeling is that since shallow networks can be made to have equivalent accuracy to deep networks, that the real challenge isn't topology but training. Hobbyists have access to so much processing…
Earlier quoted context omitted.
Because as far as I know, genetic algorithms perform poorly compared to basically every other optimization technique. I'm under the impression that they're not actually used for anything interesting these days, now that the initial hype about implementing natural selection on a computer has died down. Stochastic hill climbing as a baseline method for evaluating genetic algorithms: http://papers.nips.cc/paper/1172-sto…
Try optimizing for the shape of an antenna using hill climbing. Thing about evolutionary networks is that they can evolve topology, which usually is just a thing decided by people implementing NNs (and not guaranteed to be anywhere near most optimal). I don't see why given enough time GAs (or EAs) could not outperform simple backpropagation based on differentiating simple cost functions.
More seriously, thinking about metaheuristic, which approaches at which scale, i.e. machine learning architecture, that's the future.
Note that using genetic algorithms to come up with initial weights for neural networks was the state of the art in the late 90s/early 2000s. So this paper is not as novel as it seems, but it's good to have it for reference. My feeling is that since shallow networks can be made to have equivalent accuracy to deep networks, that the real challenge isn't topology but training. Hobbyists have access to so much processing…
Also previously on HN http://www.visiondummy.com/2014/04/curse-dimensionality-affe...
Note that using genetic algorithms to come up with initial weights for neural networks was the state of the art in the late 90s/early 2000s. So this paper is not as novel as it seems, but it's good to have it for reference. My feeling is that since shallow networks can be made to have equivalent accuracy to deep networks, that the real challenge isn't topology but training. Hobbyists have access to so much processing…
Earlier quoted context omitted.
> I don't see why given enough time GAs (or EAs) could not outperform simple backpropagation based on differentiating simple cost functions. It's already happened. You are it.
This was the line of reasoning that led to the initial hype about GAs, but I don't think it's sound. We still don't understand exactly how natural selection works, so there's no reason to believe the GAs we can implement now have the "special sauce" that makes real world natural selection effective. We don't know how to implement genotypes well, we don't know how to implement recombination or mutation well, etc.
Looking at current methods of optimizing neural networks we cannot know if we are far from the upper bound or if we've already reached it... R&D effort in this area could be a dead end.
In 2005 I took a sub-symbolic AI course where one of the homework assignments was to evolve a neural network using genetic algorithms. So that in itself is nothing new. The novelty of the paper seems to be the scale of the solution. Personally, I hope we'll see a little bit of revival of genetic algorithms/programming -- I think it's promosing in design (cars, aeroplanes, architecture, etc.), since it's good at findi…