Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Hamilton-Jacobi-Bellman Equation: Reinforcement Learning and Diffusion Models

dani2442.github.io

Post reply on HN
Print view View original

51–59 of 59 posts

Re: Hamilton-Jacobi-Bellman Equation: Reinforcement Learning and Diffusion Models

#51

Post by srean »

p1esk wrote: post #50
srean wrote: post #46

Earlier quoted context omitted.

Numerical overflow mostly no, but in case of exploding gradient, yes especially about coming up with a way to handle it, on your own, from scratch. After all, it took the research community some time to figure out a fix for that. But the examples you quoted were not my examples, at least not their primary movers (the NaNs could be caused by overflow but that overflow can have a deeper cause). The examples I gave have…

Maybe I don’t understand this data labeling issue - are you talking about imbalanced classification dataset? Are hard classes under-represented or missing labels completely?

None of those (but they could be added to the mix to complicate matters).

Consider the case that the labelers creates the labelled training set by cherry picking those examples that are easy to label. He labels many, but selects the items to label according to his preference.

First question, is this even a problem. Yes, most likely. But why ? How to fix it ? When are such fixes even possible.

Re: Hamilton-Jacobi-Bellman Equation: Reinforcement Learning and Diffusion Models

#52

Post by jmalicki »

brandonb wrote: post #32
lukko wrote: post #25

I've just started to try and learn the basics of RL and the Bellman Equation - are there any good books or resources I should look at? I think this post is beyond my current level. I'm most interested in how the equation can be implemented step by step in an ML library - worked examples would be very helpful. Thank you!

OpenAI's spinning up in deep RL is free and pretty good: https://spinningup.openai.com/en/latest/ It includes both mathematical formulas and PyTorch code. I found it a bit more practical than the Sutton & Barto book, which is a classic but doesn't cover some of the more modern methods used in deep reinforcement learning.

Cool!

It's also nice that Sutton & Barto belabors a lot of old stuff that is no longer obsessed over, and this skims through that and gets to the stuff that is much more relevant today.

Re: Hamilton-Jacobi-Bellman Equation: Reinforcement Learning and Diffusion Models

#53

Post by empiricus »

brandonb wrote: post #32
lukko wrote: post #25

I've just started to try and learn the basics of RL and the Bellman Equation - are there any good books or resources I should look at? I think this post is beyond my current level. I'm most interested in how the equation can be implemented step by step in an ML library - worked examples would be very helpful. Thank you!

OpenAI's spinning up in deep RL is free and pretty good: https://spinningup.openai.com/en/latest/ It includes both mathematical formulas and PyTorch code. I found it a bit more practical than the Sutton & Barto book, which is a classic but doesn't cover some of the more modern methods used in deep reinforcement learning.

Even this OpenAI course is from 2020? Are there no useful recent updates on the subject, especially now with everyone working and using RL?

Re: Hamilton-Jacobi-Bellman Equation: Reinforcement Learning and Diffusion Models

#54

Post by in-silico »

porridgeraisin wrote: post #42
lukko wrote: post #25

I've just started to try and learn the basics of RL and the Bellman Equation - are there any good books or resources I should look at? I think this post is beyond my current level. I'm most interested in how the equation can be implemented step by step in an ML library - worked examples would be very helpful. Thank you!

The bellman equations (exactly as written above) are not found in ML libraries. This is because they work assuming you know a model of the data. Most real world RL is model-free RL. Or, like in LLMs, "model is known but too big to practically use" RL. Apart from the resources you use (good ones in other comments already), try to get the initial mental model of the whole field right, that is important since everything…

> The bellman equations (exactly as written above) are not found in ML libraries. This is because they work assuming you know a model of the data. Most real world RL is model-free RL.

Q-learning (the usual application of the Bellman equation) is generally model-free. It is also commonly found in reinforcement learning libraries.

Re: Hamilton-Jacobi-Bellman Equation: Reinforcement Learning and Diffusion Models

#55

Post by porridgeraisin »

in-silico wrote: post #54
porridgeraisin wrote: post #42

Earlier quoted context omitted.

The bellman equations (exactly as written above) are not found in ML libraries. This is because they work assuming you know a model of the data. Most real world RL is model-free RL. Or, like in LLMs, "model is known but too big to practically use" RL. Apart from the resources you use (good ones in other comments already), try to get the initial mental model of the whole field right, that is important since everything…

> The bellman equations (exactly as written above) are not found in ML libraries. This is because they work assuming you know a model of the data. Most real world RL is model-free RL. Q-learning (the usual application of the Bellman equation) is generally model-free. It is also commonly found in reinforcement learning libraries.

Usually deep Q learning is found in libraries where you function-approximate Q with a NN, which I alluded to in one of my later paragraphs (the approximation one).

Re: Hamilton-Jacobi-Bellman Equation: Reinforcement Learning and Diffusion Models

#56

Post by p1esk »

srean wrote: post #51
p1esk wrote: post #50

Earlier quoted context omitted.

Maybe I don’t understand this data labeling issue - are you talking about imbalanced classification dataset? Are hard classes under-represented or missing labels completely?

None of those (but they could be added to the mix to complicate matters). Consider the case that the labelers creates the labelled training set by cherry picking those examples that are easy to label. He labels many, but selects the items to label according to his preference. First question, is this even a problem. Yes, most likely. But why ? How to fix it ? When are such fixes even possible.

Yes, this is a problem - the most challenging samples might not even be present in your training data. This means your model will not perform well if real world data has lots of challenging samples.

This can be partially solved if we make some assumptions about your labeller:

1. they have still picked enough challenging samples.

2. their preferences are still based on features you care about.

3. he labelled the challenging samples correctly.

And probably some other assumptions should hold for distribution of labels, etc. But what we can do in this situation is first try to model that labeller preferences, by training a binary classifier - how likely he would choose this sample for labelling from the real-world distribution? If we train that classifier, we can then assign its confidence as a sample weight when preparing our training dataset (less likely samples get more weight). This would force our main classifier to pay more attention to the challenging samples during training.

This could help somewhat if all assumptions hold, but in practice I would not expect much improvement, and the solution above can easily make it worse - this problem needs to be solved by better labelling.

How did you solve it?

Re: Hamilton-Jacobi-Bellman Equation: Reinforcement Learning and Diffusion Models

#57

Post by srean »

p1esk wrote: post #56
srean wrote: post #51

Earlier quoted context omitted.

None of those (but they could be added to the mix to complicate matters). Consider the case that the labelers creates the labelled training set by cherry picking those examples that are easy to label. He labels many, but selects the items to label according to his preference. First question, is this even a problem. Yes, most likely. But why ? How to fix it ? When are such fixes even possible.

Yes, this is a problem - the most challenging samples might not even be present in your training data. This means your model will not perform well if real world data has lots of challenging samples. This can be partially solved if we make some assumptions about your labeller: 1. they have still picked enough challenging samples. 2. their preferences are still based on features you care about. 3. he labelled the chall…

By using the (estimated) Radon Nikodym derivative between the the two measures -- the measure from which the labelers samples and the deployed to measure from which the on-deployment items are presumably sampled.

For this to work the two measures need to be absolutely continuous with each other.

This is close to your pre-penultimate paragraph and that's mathy enough. This done right can take care of bias but may do so at the expense of variance, so this Radon Nikodym derivative that is estimated needs to be done so under appropriate regularization in the function space.

Thinking of the solution in these terms requires mathematical thinking.

Now let's consider the case where some features may be missing on instances at the time of deployment but always present in training and the features are uncorrelated with each other (by construction).

Re: Hamilton-Jacobi-Bellman Equation: Reinforcement Learning and Diffusion Models

#58

Post by in-silico »

porridgeraisin wrote: post #55
in-silico wrote: post #54

Earlier quoted context omitted.

> The bellman equations (exactly as written above) are not found in ML libraries. This is because they work assuming you know a model of the data. Most real world RL is model-free RL. Q-learning (the usual application of the Bellman equation) is generally model-free. It is also commonly found in reinforcement learning libraries.

Usually deep Q learning is found in libraries where you function-approximate Q with a NN, which I alluded to in one of my later paragraphs (the approximation one).

Model-free RL doesn't mean you aren't training a model. It means that you aren't explicitly building a model of the environment's f(s,a)=(s',r) transition function, which methods like Dreamer do.

Q-learning only approximates the Q-value function, not the full state transition, so it is model-free.

Re: Hamilton-Jacobi-Bellman Equation: Reinforcement Learning and Diffusion Models

#59

Post by porridgeraisin »

in-silico wrote: post #58
porridgeraisin wrote: post #55

Earlier quoted context omitted.

Usually deep Q learning is found in libraries where you function-approximate Q with a NN, which I alluded to in one of my later paragraphs (the approximation one).

Model-free RL doesn't mean you aren't training a model. It means that you aren't explicitly building a model of the environment's f(s,a)=(s',r) transition function, which methods like Dreamer do. Q-learning only approximates the Q-value function, not the full state transition, so it is model-free.

I know that... I didn't say Q learning is not model-free.

Do you seriously think a writer of that post would think model-free RL means "not training a model at all"?

> Dreamer

Especially when I later mentioned dreamer myself specifically as a model-based algorithm

> Site guidelines: Please respond to the strongest plausible interpretation of what someone says

Post reply on HN