Hrm so are markov decision processes RL?
That's one of RL's traditional formulations, yes. Bandits problems are another one. They've been generalized together into POMDPs partially observable Markov decision processes.
Introduction to Reinforcement Learning (2015)
31–40 of 45 posts
Re: Introduction to Reinforcement Learning (2015)
#32This is (roughly) the same course but the lectures are from 2018. The audio/video quality is quite a
a bit better:
https://youtube.com/playlist?list=PLqYmG7hTraZBKeNJ-JE_eyJHZ...
Re: Introduction to Reinforcement Learning (2015)
#33Asking from a layman's perspective: I've read a bit about genetic algorithms or evolutionary computation at some point. Apparently it achieves good results as it can find discrete solutions for complex, well defined problems. Reinforcement learning is something I know even less about. But from what I gathered it is also most successful in well defined problems and systems (such as games). So my question is: How do th…
Hmm... One way that I look at it is evolutionary computation is an optimization strategy. It's characterized by tracking a population of candidates, discarding the lowest scoring, mutating the survivors, and cross-combining elements from multiple candidates.
RL is an optimization domain. It's the name of the problem, not the solution. You can straightforwardly use evolutionary algorithms on RL problems. However, a lot of the recent success in RL has come from using deep learning to try to solve various RL problems, not from trying evolutionary computation.
Re: Introduction to Reinforcement Learning (2015)
#34Asking from a layman's perspective: I've read a bit about genetic algorithms or evolutionary computation at some point. Apparently it achieves good results as it can find discrete solutions for complex, well defined problems. Reinforcement learning is something I know even less about. But from what I gathered it is also most successful in well defined problems and systems (such as games). So my question is: How do th…
Hmm... One way that I look at it is evolutionary computation is an optimization strategy. It's characterized by tracking a population of candidates, discarding the lowest scoring, mutating the survivors, and cross-combining elements from multiple candidates. RL is an optimization domain. It's the name of the problem, not the solution. You can straightforwardly use evolutionary algorithms on RL problems. However, a lo…
That makes sense thank you! It's easy to conflate the concept/problem with the tool as an outsider.
Re: Introduction to Reinforcement Learning (2015)
#35Earlier quoted context omitted.
Hmm... One way that I look at it is evolutionary computation is an optimization strategy. It's characterized by tracking a population of candidates, discarding the lowest scoring, mutating the survivors, and cross-combining elements from multiple candidates. RL is an optimization domain. It's the name of the problem, not the solution. You can straightforwardly use evolutionary algorithms on RL problems. However, a lo…
That makes sense thank you! It's easy to conflate the concept/problem with the tool as an outsider.
And it doesn't help that popular sources are perfectly happy to just call things AI or RL and leave it at that. The popular discourse around AI sometimes sounds like "How do computers work? Programming. How does google work? Programming. How do games work? Programming." Like, they're not wrong, just too vague to be useful.
Re: Introduction to Reinforcement Learning (2015)
#36Asking from a layman's perspective: I've read a bit about genetic algorithms or evolutionary computation at some point. Apparently it achieves good results as it can find discrete solutions for complex, well defined problems. Reinforcement learning is something I know even less about. But from what I gathered it is also most successful in well defined problems and systems (such as games). So my question is: How do th…
a big conceptual point in RL is the focus on the Bellman equation. value of a state equals immediate reward plus discounted future value. if you know the value of every state, just always move to pick the highest value.
well known methods like Q-learning are basically just iterative, approximate methods to find solutions to the Bellman equation — i.e. a measure of value for every state of the world, such that the Bellman equation is satisfied.
policy optimization methods don’t do this, but there are still mathematical connections back to the Bellman equation (there is a duality relationship between value functions and policies).
I would say this focus is a big part of what makes the field of RL unique.
Re: Introduction to Reinforcement Learning (2015)
#37I used to be a bit more excited about RL. I mean, it's still definitely something I have to learn, but one aspect of it _seems_ lacking to me and is messing with my motivation to learn it. I'm sure someone will happily explain all the ways I am ignorant. It seems like there is a lot of emphasis on "direct RL" or whatever where they don't even really think about the model much, but it's I guess often inside of the pol…
I don't think people have given up on model based RL, it is just that describing a proper model is (like you are saying) very difficult.
in the case you haven't seen or read the following: https://bair.berkeley.edu/blog/2019/12/12/mbpo/
Re: Introduction to Reinforcement Learning (2015)
#38Asking from a layman's perspective: I've read a bit about genetic algorithms or evolutionary computation at some point. Apparently it achieves good results as it can find discrete solutions for complex, well defined problems. Reinforcement learning is something I know even less about. But from what I gathered it is also most successful in well defined problems and systems (such as games). So my question is: How do th…
Hmm... One way that I look at it is evolutionary computation is an optimization strategy. It's characterized by tracking a population of candidates, discarding the lowest scoring, mutating the survivors, and cross-combining elements from multiple candidates. RL is an optimization domain. It's the name of the problem, not the solution. You can straightforwardly use evolutionary algorithms on RL problems. However, a lo…
While RL is about the problem it's also about the solution. Problems/Environments are formulated in a way where methods can be applied easily (i.e. Markov decision process) and thus the solutions are directly connected to the way the problem is formulated.
Deep learning is used for function approximation and is not in contrast with evolutionary computation. You can train a neutral network policy (mapping states to actions) with an evolutionary algorithm, but most of the success has come from methods that utilize the internal structure of the problem as mentioned earlier and evolutionary algorithms do not, which is what makes these optimization strategies both weak and powerful.
Re: Introduction to Reinforcement Learning (2015)
#39I used to be a bit more excited about RL. I mean, it's still definitely something I have to learn, but one aspect of it _seems_ lacking to me and is messing with my motivation to learn it. I'm sure someone will happily explain all the ways I am ignorant. It seems like there is a lot of emphasis on "direct RL" or whatever where they don't even really think about the model much, but it's I guess often inside of the pol…
One thing to keep in mind about direct (learn the policy/behavior) versus indirect (learn the model and then simulate behaviors on the model to choose the best) is that sometimes it's much easier to find a good enough policy than it is to learn an accurate enough model for simulation. Driving is a good example of this. Most of the time all you need to do is stay in your lane and obey the rules for intersections. A simulation of a driving environment, on the other hand, is quite difficult.
Re: Introduction to Reinforcement Learning (2015)
#40I used to be a bit more excited about RL. I mean, it's still definitely something I have to learn, but one aspect of it _seems_ lacking to me and is messing with my motivation to learn it. I'm sure someone will happily explain all the ways I am ignorant. It seems like there is a lot of emphasis on "direct RL" or whatever where they don't even really think about the model much, but it's I guess often inside of the pol…
You do not necessarily need to fully know the environment you are in, but you need to be able to evaluate how good the actions that you can take are in terms of an utility function. That’s how a RL algorithm can learn that going through a wall is a bad decision (reward(“ahead”) “$0”).
I think the main problem with RL is deciding if an utility function — as precise as it may be — can fully capture/estimate all nuances of an environment. Another problem is at adapting to the environment by having new actions added dynamically into your model and having it to converge as quickly as possible.