Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Hamilton-Jacobi-Bellman Equation: Reinforcement Learning and Diffusion Models

dani2442.github.io

Post reply on HN
Print view View original

21–30 of 59 posts

Re: Hamilton-Jacobi-Bellman Equation: Reinforcement Learning and Diffusion Models

#21

Post by shiandow »

measurablefunc wrote: post #2

It's not clear or obvious why continuous semantics should be applicable on a digital computer. This might seem like nitpicking but it's not, there is a fundamental issue that is always swept under the rug in these kinds of analysis which is about reconciling finitary arithmetic over bit strings & the analytical equations which only work w/ infinite precision over the real or complex numbers as they are usually define…

It is definitely not obvious, but I wouldn't say it is completely unclear.

For instance we know that algorithms like the leapfrog integrator not only approximate a physical system quite well but even conserve the energy, or rather a quantity that approximates the true energy.

There are plenty of theorems about the accuracy and other properties of numerical algorithms.

Re: Hamilton-Jacobi-Bellman Equation: Reinforcement Learning and Diffusion Models

#22

Post by griffzhowl »

rsp1984 wrote: post #17
lain98 wrote: post #16

I find myself completely outclassed by mathematicians in my own field. I tried to learn a little math on the side after my regular software engineer gig but I'm completely outclassed by phd's. I am unsure of the next course of action or if software will survive another 5 years and how my career will look like in the future. Seems like I am engaged in the ice trade and they are about to invent the refrigerator.

Don't despair. The key to becoming proficient in advanced subjects like this one is to first try to understand the fundamentals in plain language and pictures in your mind. Ignore the equations. Ask AI to explain the topic at hand at the most fundamental level. Once the fundamental concepts are understood, what problem is being solved and where the key difficulties are, only then the equations will start to make sens…

> People who really grasp a subject can usually explain it well in plain language.

That's very much a matter of style. An equation is often the plainest way of expressing something

Re: Hamilton-Jacobi-Bellman Equation: Reinforcement Learning and Diffusion Models

#24

Post by sebzuddas »

Cloudly wrote: post #4

Ever since the control bug bit me in my EE undergrad years I am happy to see how useful the knowledge remains. Of course the underlying math of optimization remains general but the direct applications of control theory made it much more appetizing for me to struggle through.

My favourite subject!

Re: Hamilton-Jacobi-Bellman Equation: Reinforcement Learning and Diffusion Models

#25

Post by lukko »

I've just started to try and learn the basics of RL and the Bellman Equation - are there any good books or resources I should look at? I think this post is beyond my current level.

I'm most interested in how the equation can be implemented step by step in an ML library - worked examples would be very helpful.

Thank you!

Re: Hamilton-Jacobi-Bellman Equation: Reinforcement Learning and Diffusion Models

#26

Post by RA_Fisher »

lain98 wrote: post #16

I find myself completely outclassed by mathematicians in my own field. I tried to learn a little math on the side after my regular software engineer gig but I'm completely outclassed by phd's. I am unsure of the next course of action or if software will survive another 5 years and how my career will look like in the future. Seems like I am engaged in the ice trade and they are about to invent the refrigerator.

AI makes it easier to catch up. :)

Re: Hamilton-Jacobi-Bellman Equation: Reinforcement Learning and Diffusion Models

#27

Post by ActivePattern »

lukko wrote: post #25

I've just started to try and learn the basics of RL and the Bellman Equation - are there any good books or resources I should look at? I think this post is beyond my current level. I'm most interested in how the equation can be implemented step by step in an ML library - worked examples would be very helpful. Thank you!

Reinforcement Learning by Sutton & Barto is an excellent introduction by two of the founders of the field.

Read here: http://incompleteideas.net/book/the-book-2nd.html

Re: Hamilton-Jacobi-Bellman Equation: Reinforcement Learning and Diffusion Models

#28

Post by tsimionescu »

phreeza wrote: post #6
measurablefunc wrote: post #2

It's not clear or obvious why continuous semantics should be applicable on a digital computer. This might seem like nitpicking but it's not, there is a fundamental issue that is always swept under the rug in these kinds of analysis which is about reconciling finitary arithmetic over bit strings & the analytical equations which only work w/ infinite precision over the real or complex numbers as they are usually define…

Doesn't continuous time basically mean "this is what we expect for sufficiently small time steps"? Very similar to how one would for example take the first order Taylor dynamics and use them for "sufficiently small" perturbations from equilibrium. Is there any other magic to continuous time systems that one would not expect to be solved by sufficiently small time steps?

Infinity has properties that finite approximations of it just don't have, and this can lead to serious problems for certain theorems. In the general case, the integral of a continuous function can be arbitrarily different from the sum of a finite sequence of points sampled from that function, regardless of how many points you sample - and it's even possible that the discrete version is divergent even if the continous one is convergent.

I'm not saying that this is the case here, but there generally needs to be some justification to say that a certain result that is proven for a continuous function also holds for some discrete version of it.

For a somewhat famous real-world example, it's not currently known how to produce a version of QM/QFT that works with discrete spacetime coordinates, the attempted discretizations fail to maintain the properties of the continuous equations.

Re: Hamilton-Jacobi-Bellman Equation: Reinforcement Learning and Diffusion Models

#29

Post by srean »

lukko wrote: post #25

I've just started to try and learn the basics of RL and the Bellman Equation - are there any good books or resources I should look at? I think this post is beyond my current level. I'm most interested in how the equation can be implemented step by step in an ML library - worked examples would be very helpful. Thank you!

I would recommend that you start with one of the classics (not much of deep RL)

https://www.andrew.cmu.edu/course/10-703/textbook/BartoSutto...

This will have a gentler learning curve. After this you can move on to more advanced material.

The other resource I will recommend is everything by Bertsekas. In this context, his books on dynamic programming and neurodyanamic programming.

Happy reading.

Re: Hamilton-Jacobi-Bellman Equation: Reinforcement Learning and Diffusion Models

#30

Post by srean »

numbers_guy wrote: post #20
lain98 wrote: post #16

I find myself completely outclassed by mathematicians in my own field. I tried to learn a little math on the side after my regular software engineer gig but I'm completely outclassed by phd's. I am unsure of the next course of action or if software will survive another 5 years and how my career will look like in the future. Seems like I am engaged in the ice trade and they are about to invent the refrigerator.

I guess I have the opposite experience. I have a post-graduate level of mathematical education and I am dismayed at how little there is to be gained from it, when it comes to AI/ML. Diffusion Models and Geometric Deep Learning are the only two fields where there's any math at all. Many math grads are struggling to find a job at all. They aren't outclassing programmers with their leet math skillz.

Don't worry when stochastic grads get stuck math grads get going.

(One of) The value(s) that a math grad brings is debugging and fixing these ML models when training fails. Many would not have an idea about how to even begin debugging why the trained model is not working so well, let alone how to explore fixes.

Post reply on HN