Evolution Strategies as a Scalable Alternative to Reinforcement Learning
1–10 of 27 posts
Re: Evolution Strategies as a Scalable Alternative to Reinforcement Learning
#2Policy search is very effective. This is not a new finding as this article seems to suggest in the abstract.
Variants of ES have been used for years (http://dl.acm.org/citation.cfm?id=1645634). The article seems to ignore almost all the work in robotics, e.g. from Jan Peters' research group (http://www.jan-peters.net/ -> publications).
The good thing is that we have one more paper that justifies this research direction and a little bit more public attention.
Re: Evolution Strategies as a Scalable Alternative to Reinforcement Learning
#3I'm not an expert but for me it's seems that the proposed approach doesn't compute the value function . By optimizing directly the policy function aren't we losing some key ingredient for generalization? I mean could this algorithm be used in a fully non deterministic environment ? Like human vs machine?
Re: Evolution Strategies as a Scalable Alternative to Reinforcement Learning
#4I feel like this is more an argument against the efficiency of flat (non-hierarchical, not model-based) Deep Reinforcement Learning than an argument for Genetic/Evolutionary Algorithms. As in, if you're not solving a task more efficiently than Evolutionary Algorithms or more difficult than they can handle, you're not doing something right. Similar to how Deep PCA and Deep Random Forests blew ConvNets out of the water on basic benchmarks like MNIST, but couldn't compete on larger datasets, indicated the proof we should require before getting excited about a new technique.
Re: Evolution Strategies as a Scalable Alternative to Reinforcement Learning
#5I feel like this is more an argument against the efficiency of flat (non-hierarchical, not model-based) Deep Reinforcement Learning than an argument for Genetic/Evolutionary Algorithms. As in, if you're not solving a task more efficiently than Evolutionary Algorithms or more difficult than they can handle, you're not doing something right. Similar to how Deep PCA and Deep Random Forests blew ConvNets out of the water…
Deep PCA: https://arxiv.org/abs/1404.3606
Deep Forest: https://arxiv.org/abs/1702.08835
Re: Evolution Strategies as a Scalable Alternative to Reinforcement Learning
#6I feel like this is more an argument against the efficiency of flat (non-hierarchical, not model-based) Deep Reinforcement Learning than an argument for Genetic/Evolutionary Algorithms. As in, if you're not solving a task more efficiently than Evolutionary Algorithms or more difficult than they can handle, you're not doing something right. Similar to how Deep PCA and Deep Random Forests blew ConvNets out of the water…
Not sure where you're getting this from, but CNNs are definitely still state of the art on MNIST. Papers often cite outdated CNN numbers; in fact the best published CNN accuracy numbers are probably lower than they could be - vanilla MNIST for supervised learning is a pretty useless benchmark for computer vision researchers now.
Re: Evolution Strategies as a Scalable Alternative to Reinforcement Learning
#7I feel like this is more an argument against the efficiency of flat (non-hierarchical, not model-based) Deep Reinforcement Learning than an argument for Genetic/Evolutionary Algorithms. As in, if you're not solving a task more efficiently than Evolutionary Algorithms or more difficult than they can handle, you're not doing something right. Similar to how Deep PCA and Deep Random Forests blew ConvNets out of the water…
Not sure where you're getting this from, but CNNs are definitely still state of the art on MNIST. Papers often cite outdated CNN numbers; in fact the best published CNN accuracy numbers are probably lower than they could be - vanilla MNIST for supervised learning is a pretty useless benchmark for computer vision researchers now.
My bad, I should have been more specific. The Deep PCA paper shows superior performance not on baseline MNIST, but on the MNIST variations.
I also agree that vanilla MNIST is pretty useless for Computer Vision researches and was trying (awkwardly) to support that idea by showing how these other non Deep Learning techniques performed equally well.
Re: Evolution Strategies as a Scalable Alternative to Reinforcement Learning
#8I'm not an expert but for me it's seems that the proposed approach doesn't compute the value function . By optimizing directly the policy function aren't we losing some key ingredient for generalization? I mean could this algorithm be used in a fully non deterministic environment ? Like human vs machine?
I am not sure at the moment. I guess the main problem of ES in a complex non-deterministic environment is that it would average over multiple local minima that occur in one generation which would result in a non-optimum solution. There are policy search methods that address this problem (e.g. VIPS https://scholar.google.com/citations?view_op=view_citation&h... , hierarchichal REPS https://scholar.google.com/citations?view_op=view_citation&h...).
Temporal credit assignment is another problem: the policy is updated after a full episode and there is no way to use the information which step was responsible for which reward. Policy search usually works well if the value function is very complex and the optimal policy is simple.
Re: Evolution Strategies as a Scalable Alternative to Reinforcement Learning
#9This is a fantastic summary of both RL and ES (know as genetoc algorithms too). Kudos to authors!
Re: Evolution Strategies as a Scalable Alternative to Reinforcement Learning
#10Great write up.
I had a similar example in my 20 year old book 'C++ Power Paradigms' in which I used a genetic algorithm to train the weights in a recurrent network. As a performance hack, weights were initially represented by just a few bits, and the bit length would gradually be increased, which greatly increased the search space. I never got this to scale past small networks, but I have thought of revisiting my old code since I have a lot more computing power available now, compared to 20 years ago.