Reinforcement learning is supervised learning on optimized data
bair.berkeley.edu
Reinforcement learning is supervised learning on optimized data
1–10 of 19 posts
Re: Reinforcement learning is supervised learning on optimized data
#2Nice definition of “school”, which is the most efficient method to teach/learn we have.
Re: Reinforcement learning is supervised learning on optimized data
#3This seems like a HUGE insight! As I understand it, they show that RL can effectively be recast as two sub-problems:
1. learning a policy that imitates your own behavior on prior experience, which is a trivial supervised learning problem
2. learning how to weight the importance of prior experiences (learning a data distribution), for which the authors have derived a lower bound
Given a pool of experience, this seems like a fantastic off-policy method to optimize arbitrary reward functions. The main shortcomings I see with this method is that it still does not lead to any significant insights into how to collect new data online, which is a major open problem in RL.
Re: Reinforcement learning is supervised learning on optimized data
#4Nice definition of “school”, which is the most efficient method to teach/learn we have.
Efficient for those doing the teaching, but not for the one being taught.
Re: Reinforcement learning is supervised learning on optimized data
#5This seems like a HUGE insight! As I understand it, they show that RL can effectively be recast as two sub-problems: 1. learning a policy that imitates your own behavior on prior experience, which is a trivial supervised learning problem 2. learning how to weight the importance of prior experiences (learning a data distribution), for which the authors have derived a lower bound Given a pool of experience, this seems…
> I'm also wondering why the authors didn't publish any experiments to show that it works...
This is a blog post. It cites three of the authors' papers that each contain empirical results. The abstract of the first ends:
"We formally show that this iterated supervised learning procedure optimizes a bound on the RL objective, derive performance bounds of the learned policy, and empirically demonstrate improved goal-reaching performance and robustness over current RL algorithms in several benchmark tasks."
Re: Reinforcement learning is supervised learning on optimized data
#6I had a hard time understanding how Actor-Critic methods relate to this insight. Anyone care to explain?
Re: Reinforcement learning is supervised learning on optimized data
#7Re: Reinforcement learning is supervised learning on optimized data
#8It seems to me that they are basically describing a variational formulation of the "optimization perspective" of reinforcement learning, which is cool, but I am confused... where is the supervised learning? Like what is the input and what is the output?
Re: Reinforcement learning is supervised learning on optimized data
#9Although the "data problem" is already well known by all ML and RL engineers/researchers (with varying degree depending on what one works on), it's an innovative and concise post that really boils it down. It's particularly important for modeling and when you have to generate all the data on your own.
Re: Reinforcement learning is supervised learning on optimized data
#10This seems like a HUGE insight! As I understand it, they show that RL can effectively be recast as two sub-problems: 1. learning a policy that imitates your own behavior on prior experience, which is a trivial supervised learning problem 2. learning how to weight the importance of prior experiences (learning a data distribution), for which the authors have derived a lower bound Given a pool of experience, this seems…
> I'm also wondering why the authors didn't publish any experiments to show that it works... This is a blog post. It cites three of the authors' papers that each contain empirical results. The abstract of the first ends: "We formally show that this iterated supervised learning procedure optimizes a bound on the RL objective, derive performance bounds of the learned policy, and empirically demonstrate improved goal-re…
> empirically demonstrate improved goal-reaching performance and robustness over current RL algorithms
It's interesting that their choice of current algorithms includes PPO but not e.g. Deepmind's Rainbow agent that achieved state of the art performance on many measures: https://arxiv.org/abs/1710.02298