Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Hamilton-Jacobi-Bellman Equation: Reinforcement Learning and Diffusion Models

dani2442.github.io

Post reply on HN
Print view View original

31–40 of 59 posts

Re: Hamilton-Jacobi-Bellman Equation: Reinforcement Learning and Diffusion Models

#31

Post by p1esk »

srean wrote: post #30
numbers_guy wrote: post #20

Earlier quoted context omitted.

I guess I have the opposite experience. I have a post-graduate level of mathematical education and I am dismayed at how little there is to be gained from it, when it comes to AI/ML. Diffusion Models and Geometric Deep Learning are the only two fields where there's any math at all. Many math grads are struggling to find a job at all. They aren't outclassing programmers with their leet math skillz.

Don't worry when stochastic grads get stuck math grads get going. (One of) The value(s) that a math grad brings is debugging and fixing these ML models when training fails. Many would not have an idea about how to even begin debugging why the trained model is not working so well, let alone how to explore fixes.

Debugging ML models (large part of my job) requires very little math. Engineering experience and mindset is a lot more relevant for debugging. Complicated math is typically needed when you want invent new loss functions, or new methods for regularization, normalization or model compression.

Re: Hamilton-Jacobi-Bellman Equation: Reinforcement Learning and Diffusion Models

#32

Post by brandonb »

lukko wrote: post #25

I've just started to try and learn the basics of RL and the Bellman Equation - are there any good books or resources I should look at? I think this post is beyond my current level. I'm most interested in how the equation can be implemented step by step in an ML library - worked examples would be very helpful. Thank you!

OpenAI's spinning up in deep RL is free and pretty good: https://spinningup.openai.com/en/latest/

It includes both mathematical formulas and PyTorch code.

I found it a bit more practical than the Sutton & Barto book, which is a classic but doesn't cover some of the more modern methods used in deep reinforcement learning.

Re: Hamilton-Jacobi-Bellman Equation: Reinforcement Learning and Diffusion Models

#33

Post by measurablefunc »

phreeza wrote: post #13
measurablefunc wrote: post #11

Earlier quoted context omitted.

Point still stands, translation from continuous to discrete is not as simple as people think.

Numerical issues totally exist but the reason has nothing to do with the fact that Cauchy sequences don't exist on a computer imo.

The abstract formulation is different from the concrete implementation. It is precisely b/c the abstractions do not exist on computers that the abstract analysis does not automatically transfer the necessary analytical properties to the digital implementation. Cauchy sequences & Dedekind cuts are abstract & do not exist on digital computers.

Re: Hamilton-Jacobi-Bellman Equation: Reinforcement Learning and Diffusion Models

#34

Post by measurablefunc »

shiandow wrote: post #21
measurablefunc wrote: post #2

It's not clear or obvious why continuous semantics should be applicable on a digital computer. This might seem like nitpicking but it's not, there is a fundamental issue that is always swept under the rug in these kinds of analysis which is about reconciling finitary arithmetic over bit strings & the analytical equations which only work w/ infinite precision over the real or complex numbers as they are usually define…

It is definitely not obvious, but I wouldn't say it is completely unclear. For instance we know that algorithms like the leapfrog integrator not only approximate a physical system quite well but even conserve the energy, or rather a quantity that approximates the true energy. There are plenty of theorems about the accuracy and other properties of numerical algorithms.

How do they apply in this case?

Re: Hamilton-Jacobi-Bellman Equation: Reinforcement Learning and Diffusion Models

#35

Post by srean »

p1esk wrote: post #31
srean wrote: post #30

Earlier quoted context omitted.

Don't worry when stochastic grads get stuck math grads get going. (One of) The value(s) that a math grad brings is debugging and fixing these ML models when training fails. Many would not have an idea about how to even begin debugging why the trained model is not working so well, let alone how to explore fixes.

Debugging ML models (large part of my job) requires very little math. Engineering experience and mindset is a lot more relevant for debugging. Complicated math is typically needed when you want invent new loss functions, or new methods for regularization, normalization or model compression.

You are perhaps talking about some simple plumbing bugs. There are other kinds:

Why didn't the training converge

Validation/test errors are great but why is performance in the wild so poor

Why is the model converging so soon

Why is this all zero

Why is this NaN

Model performance is not great, do I need to move to something more complicated or am I doing something wrong

Did the nature of the upstream data change ?

Sometimes this feature is missing, how should I deal with this

The training set and the data on which the model will be deployed are different. How to address this problem

The labelers labelled only the instances that are easy to label, not chosen uniformly from the data. How to train with such skewed label selection

I need to update model but with a few thousand data points but not train from scratch. How do I do it

Model too large which doubles can I replace with float32

So on and so forth. Many times models are given up on prematurely because the expertise to investigate lackluster performance does not exist in the team.

Re: Hamilton-Jacobi-Bellman Equation: Reinforcement Learning and Diffusion Models

#36

Post by sardukardboard »

lukko wrote: post #25

I've just started to try and learn the basics of RL and the Bellman Equation - are there any good books or resources I should look at? I think this post is beyond my current level. I'm most interested in how the equation can be implemented step by step in an ML library - worked examples would be very helpful. Thank you!

I worked thru David Silver’s RL course a while back, it’s got great explanations as he builds up the equations. It’s light on implementation, but the intuitive side really complements more code-heavy examples that lack the “why” behind the equations.

https://davidstarsilver.wordpress.com/teaching/

Re: Hamilton-Jacobi-Bellman Equation: Reinforcement Learning and Diffusion Models

#37

Post by porridgeraisin »

numbers_guy wrote: post #20
lain98 wrote: post #16

I find myself completely outclassed by mathematicians in my own field. I tried to learn a little math on the side after my regular software engineer gig but I'm completely outclassed by phd's. I am unsure of the next course of action or if software will survive another 5 years and how my career will look like in the future. Seems like I am engaged in the ice trade and they are about to invent the refrigerator.

I guess I have the opposite experience. I have a post-graduate level of mathematical education and I am dismayed at how little there is to be gained from it, when it comes to AI/ML. Diffusion Models and Geometric Deep Learning are the only two fields where there's any math at all. Many math grads are struggling to find a job at all. They aren't outclassing programmers with their leet math skillz.

The real use is in actually seeing connections. Every field has their own maths and their own terminologies, their own assumptions for theorems, etc.

More often than not this is duplicated work (mathematically speaking) and there is a lot to be gained by sharing advances in either field by running it through a "translation". This has happened many times historically - a lot of the "we met at a cafe and worked it out on a napkin" inventions are exactly that.

Math proficiency helps a lot at that. The level of abstraction you deal with is naturally high.

Recently, the problem of actually knowing every field enough, just cursorily, to make connections is easier with AI. Modern LLMs do approximate retrieval and still need a planner + verifier, the mathematician can be that.

This is somewhat adjacent to what terry tao spoke about, and the setup is sort of what alpha evolve does.

You get that impression because such advances are high impact and rare (because they are difficult). Most advances come as a sequence of field-specific assumption, field-specific empirical observation, field-specific theorem, and so on. We only see the advances that are actually made, leading to an observation bias.

Re: Hamilton-Jacobi-Bellman Equation: Reinforcement Learning and Diffusion Models

#38

Post by ecshafer »

lain98 wrote: post #16

I find myself completely outclassed by mathematicians in my own field. I tried to learn a little math on the side after my regular software engineer gig but I'm completely outclassed by phd's. I am unsure of the next course of action or if software will survive another 5 years and how my career will look like in the future. Seems like I am engaged in the ice trade and they are about to invent the refrigerator.

IMO Computer Science doesn't have enough mathematics in the core curriculum. I think more CS students should be double majoring or minoring in Physics and/or Math. The skills you gain in analyzing problems and constructing models in Physics, finding truth/false values and analyzing problems in math, and the algorithmic skills in CS really compliment each other.

Instead of people "hacking" university education to make them purely fotm job training centers. The real hack would be something that really drills down at the fundamentals. CS, Math, Physics, and Philosophy to get an all around education in approaching problems from fundamentals I think would be the optimal school experience.

Re: Hamilton-Jacobi-Bellman Equation: Reinforcement Learning and Diffusion Models

#39

Post by swimmingbrain »

lukko wrote: post #25

I've just started to try and learn the basics of RL and the Bellman Equation - are there any good books or resources I should look at? I think this post is beyond my current level. I'm most interested in how the equation can be implemented step by step in an ML library - worked examples would be very helpful. Thank you!

[dead]

Re: Hamilton-Jacobi-Bellman Equation: Reinforcement Learning and Diffusion Models

#40

Post by swimmingbrain »

lain98 wrote: post #16

I find myself completely outclassed by mathematicians in my own field. I tried to learn a little math on the side after my regular software engineer gig but I'm completely outclassed by phd's. I am unsure of the next course of action or if software will survive another 5 years and how my career will look like in the future. Seems like I am engaged in the ice trade and they are about to invent the refrigerator.

[dead]
Post reply on HN