Reinforcement Learning as One Big Sequence Modeling Problem
trajectory-transformer.github.io
Reinforcement Learning as One Big Sequence Modeling Problem
1–10 of 10 posts
Re: Reinforcement Learning as One Big Sequence Modeling Problem
#2A very interesting recent interview with an author of the concurrent work “decision transformer”
https://open.spotify.com/episode/5uYwk4KcqRtdMjn7hlVOZD?si=2...
Re: Reinforcement Learning as One Big Sequence Modeling Problem
#3Here’s a more recent work: https://arxiv.org/abs/2205.14953
Note: online training.
Re: Reinforcement Learning as One Big Sequence Modeling Problem
#4A very interesting recent interview with an author of the concurrent work “decision transformer” https://open.spotify.com/episode/5uYwk4KcqRtdMjn7hlVOZD?si=2...
Can you explain the main differences between DT and the submitted paper?
DT is getting all the attention (ha! pun!)... maybe it is faster
EDIT: ah. Here is a nice quote:
Decision Transformer [5] bypasses the need for computing cumulative rewards through dynamic programming, but rather generates future actions conditioning on the desired returns, past states and actions
Re: Reinforcement Learning as One Big Sequence Modeling Problem
#5Here’s a more recent work: https://arxiv.org/abs/2205.14953 Note: online training.
Is there a simpler paper that addresses online RL online with the recent decision transformer / sequence model?
The multi agent aspect here really complicates things
Re: Reinforcement Learning as One Big Sequence Modeling Problem
#6"and investigate how much of the usual machinery of reinforcement learning algorithms can be replaced with the tools..."
These sounds as if the authors have confidently figured out that the current reinforcement learning formulation is not good enough.
On the other hand, I think the recent large language models have showed us that much of the world knowledge is indeed predictive. That, if you can predict accurately (next words), you can understand higher more abstract things. The hypothesis that much of world knowledge is predictive, is very important in the framework of reinforcement learning because that means that with enough General Value functions learned off-policy, one can predict almost anything about the world that is useful to the agent in achieving its goals. (cf the Horde paper).
Re: Reinforcement Learning as One Big Sequence Modeling Problem
#7Here’s a more recent work: https://arxiv.org/abs/2205.14953 Note: online training.
Is there a simpler paper that addresses online RL online with the recent decision transformer / sequence model? The multi agent aspect here really complicates things
Online Decision Transformer
https://arxiv.org/abs/2202.05607
A very recent follow up paper on DT, addressing the limitations of the conditioning on reward, but still offline:
https://arxiv.org/abs/2205.15967
Reddit discussion:
https://reddit.com/r/MachineLearning/comments/v2sul7/r_you_c...
Re: Reinforcement Learning as One Big Sequence Modeling Problem
#8"and investigate how much of the usual machinery of reinforcement learning algorithms can be replaced with the tools..." These sounds as if the authors have confidently figured out that the current reinforcement learning formulation is not good enough. On the other hand, I think the recent large language models have showed us that much of the world knowledge is indeed predictive. That, if you can predict accurately (…
> much of the world knowledge is indeed predictive ... if you can predict accurately (next words), you can understand higher more abstract things
What do you mean with "understand"? And why are you calling "knowledge" that which is predictive?
Re: Reinforcement Learning as One Big Sequence Modeling Problem
#9"and investigate how much of the usual machinery of reinforcement learning algorithms can be replaced with the tools..." These sounds as if the authors have confidently figured out that the current reinforcement learning formulation is not good enough. On the other hand, I think the recent large language models have showed us that much of the world knowledge is indeed predictive. That, if you can predict accurately (…
> much of the world knowledge is indeed predictive ... if you can predict accurately (next words), you can understand higher more abstract things What do you mean with "understand"? And why are you calling "knowledge" that which is predictive?
> What do you mean with "understand"?
By "understand" I mean the knowledge that is missing to the agent in order to control the environment toward achieving its goal. Reinforcement learning is concerned with this sort of interaction, between an agent (a decision maker) and an (unknown) environment. The (only) goal of the agent is to maximize its cumulative sum of reward (cf the Reward hypothesis(.
> And why are you calling "knowledge" that which is predictive?
No, I do not think I am saying that, but if it comes across like that, let me be more precise.
I mean that most (but not all) of the world knowledge is predictive. An example of not-predictive knowledge is factual knowledge, like mathematics. Knowledge being predictive is important for an autonomous decision maker because the knowledge can be verified solely by the agent (not a teacher, as it is in the framework of supervised learning). One crucial thing to understand about the framework of reinforcement learning is that it makes the agent solely responsible for its way of behaving. As it should be. Then, an effective way to do that in a scalable way (to not rely on some oracle teacher or anything else) is to be able to verify any knowledge that the agent wants to acquire, in order to achieve its goals.
Re: Reinforcement Learning as One Big Sequence Modeling Problem
#10Earlier quoted context omitted.
Is there a simpler paper that addresses online RL online with the recent decision transformer / sequence model? The multi agent aspect here really complicates things
Online Decision Transformer https://arxiv.org/abs/2202.05607 A very recent follow up paper on DT, addressing the limitations of the conditioning on reward, but still offline: https://arxiv.org/abs/2205.15967 Reddit discussion: https://reddit.com/r/MachineLearning/comments/v2sul7/r_you_c...
If anyone is interested in further reading, I've put a DT bibliography at https://www.gwern.net/docs/reinforcement-learning/model/deci...