11–20 of 27 posts
This is a fantastic summary of both RL and ES (know as genetoc algorithms too). Kudos to authors!
I'm not an expert but for me it's seems that the proposed approach doesn't compute the value function . By optimizing directly the policy function aren't we losing some key ingredient for generalization? I mean could this algorithm be used in a fully non deterministic environment ? Like human vs machine?
This is a fantastic summary of both RL and ES (know as genetoc algorithms too). Kudos to authors!
While they're in the same general family of black box optimization algorithms, ES and GAs are not really the same, and in fact were developed independently. The historical roots of both are actually quite interesting.
"A Field Guide to Genetic Programming" ( https://www.amazon.com/Field-Guide-Genetic-Programming/dp/14... ) is one of the better books I've read on the matter (but I've yet to read anything by Koza). W. Langdon's webpage ( http://www0.cs.ucl.ac.uk/staff/W.Langdon/ ) has a lot of great information.
This is a fantastic summary of both RL and ES (know as genetoc algorithms too). Kudos to authors!
While they're in the same general family of black box optimization algorithms, ES and GAs are not really the same, and in fact were developed independently. The historical roots of both are actually quite interesting.
[0] http://stackoverflow.com/questions/7787232/difference-betwee...
2. Tie future awards to action at this timestep...
Can anyone help with better nitty-gritty explanation ?
Reading the code; I dont get get: 1. How to model the different actions 2. Tie future awards to action at this timestep... Can anyone help with better nitty-gritty explanation ?
The behavior of agents is determined by a "policy function". This function takes in inputs (e.g. what the agent sees) and outputs actions (e.g. what the agent does). The policy function has a set of internal parameters that determines the precise mapping from inputs to outputs.
In their work, they used a neural network as the policy function. The parameters are just all the weights of the network.
In a simple version, you start with some random weights for the NN. Then you make many copies of the network, each with a slight random variation made to the weights. For each of these altered networks, you use them to control an agent for a while, and see how well the agent performs during that trial period. Based on how well the different variations do during their trial runs, you adjust the weights of the network a small amount. You adjust the weights to be more similar to the variations that did well. Then you repeat the process indefinitely (generate new variations, test them, etc.).
"A Field Guide to Genetic Programming" ( https://www.amazon.com/Field-Guide-Genetic-Programming/dp/14... ) is one of the better books I've read on the matter (but I've yet to read anything by Koza). W. Langdon's webpage ( http://www0.cs.ucl.ac.uk/staff/W.Langdon/ ) has a lot of great information.
In the end, it doesn't matter that much which approach is taken because it's all classification problems. We just need the solution matrix, and ideally what computation went into solving it. I feel that this simple fact is lost amidst the complexity of how ML is taught today.
ML isn’t accelerating because of better code or research breakthroughs either. It’s happening because the big CPU manufacturers didn’t do anything for 20 years and GPU manufactures had their lunch. ML is straightforward, even trivial in some cases with effectively unlimited cores and bandwidth. We’re just rediscovering parallelization algorithms that were well known in functional programming generations ago. These discoveries are inevitable in a suitable playground.
I used to have this fantasy that I would get ahead of the curve enough to be able to dabble in the last human endeavor but I'm beginning to realize that that's probably never going to happen. Machines will soon beat humans in pretty much every category, and not because someone figures out how to make it all work, but because there simply isn't enough time to stop it now. There are a dozen teams around the world racing to solve any problem and anyone’s odds of being first are perhaps 10% at best. Compounded with darwinian capitalism, the risk/reward equation is headed towards infinity so fast that it’s looking like the smartest move is not to play.
Barring a dystopian future or cataclysm, I give us 10 years, certainly no more than 20, before computers can do anything people can do, at least economically. And the really eerie thing is that that won’t be the most impressive thing happening, because kids will know it’s all just hill climbing and throwing hardware at problems. It will be all the other associated technologies that come about as people abandon the old hard ways of doing things.
"A Field Guide to Genetic Programming" ( https://www.amazon.com/Field-Guide-Genetic-Programming/dp/14... ) is one of the better books I've read on the matter (but I've yet to read anything by Koza). W. Langdon's webpage ( http://www0.cs.ucl.ac.uk/staff/W.Langdon/ ) has a lot of great information.