Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Hamilton-Jacobi-Bellman Equation: Reinforcement Learning and Diffusion Models

dani2442.github.io

Post reply on HN
Print view View original

11–20 of 59 posts

Re: Hamilton-Jacobi-Bellman Equation: Reinforcement Learning and Diffusion Models

#11

Post by measurablefunc »

phyalow wrote: post #10
measurablefunc wrote: post #9

Earlier quoted context omitted.

The translation from continuous to discrete is not automatic. There is a missing verification in the linked analysis. The mapping must be verified for stability for the proper class of initial/boundary conditions. Increasing the resolution from 64 bit floats to 128 bit floats doesn't automatically give you a stable discretized optimizer from a continuous formulation.

Or you can just try stuff and see if it works

Point still stands, translation from continuous to discrete is not as simple as people think.

Re: Hamilton-Jacobi-Bellman Equation: Reinforcement Learning and Diffusion Models

#12

Post by heyethan »

phreeza wrote: post #6
measurablefunc wrote: post #2

It's not clear or obvious why continuous semantics should be applicable on a digital computer. This might seem like nitpicking but it's not, there is a fundamental issue that is always swept under the rug in these kinds of analysis which is about reconciling finitary arithmetic over bit strings & the analytical equations which only work w/ infinite precision over the real or complex numbers as they are usually define…

Doesn't continuous time basically mean "this is what we expect for sufficiently small time steps"? Very similar to how one would for example take the first order Taylor dynamics and use them for "sufficiently small" perturbations from equilibrium. Is there any other magic to continuous time systems that one would not expect to be solved by sufficiently small time steps?

[flagged]

Re: Hamilton-Jacobi-Bellman Equation: Reinforcement Learning and Diffusion Models

#13

Post by phreeza »

measurablefunc wrote: post #11
phyalow wrote: post #10

Earlier quoted context omitted.

Or you can just try stuff and see if it works

Point still stands, translation from continuous to discrete is not as simple as people think.

Numerical issues totally exist but the reason has nothing to do with the fact that Cauchy sequences don't exist on a computer imo.

Re: Hamilton-Jacobi-Bellman Equation: Reinforcement Learning and Diffusion Models

#14

Post by cubefox »

measurablefunc wrote: post #2

It's not clear or obvious why continuous semantics should be applicable on a digital computer. This might seem like nitpicking but it's not, there is a fundamental issue that is always swept under the rug in these kinds of analysis which is about reconciling finitary arithmetic over bit strings & the analytical equations which only work w/ infinite precision over the real or complex numbers as they are usually define…

Real numbers mostly appear in calculus (e.g. the chain rule in gradient descent/backpropagation), but "discrete calculus" is then used as an approximation of infinitesimal calculus. It uses "finite differences" rather than derivatives, which doesn't require real numbers:

https://en.wikipedia.org/wiki/Finite_difference

I'm not sure about applications of real numbers outside of calculus, and how to replace them there.

Re: Hamilton-Jacobi-Bellman Equation: Reinforcement Learning and Diffusion Models

#15

Post by sfpotter »

measurablefunc wrote: post #2

It's not clear or obvious why continuous semantics should be applicable on a digital computer. This might seem like nitpicking but it's not, there is a fundamental issue that is always swept under the rug in these kinds of analysis which is about reconciling finitary arithmetic over bit strings & the analytical equations which only work w/ infinite precision over the real or complex numbers as they are usually define…

This is what the field of numerical analysis exists for. These details definitely have been treated, but this was done mainly early in the field's history; for example, by people like Wilkinson and Kahan...

Re: Hamilton-Jacobi-Bellman Equation: Reinforcement Learning and Diffusion Models

#16

Post by lain98 »

I find myself completely outclassed by mathematicians in my own field. I tried to learn a little math on the side after my regular software engineer gig but I'm completely outclassed by phd's.

I am unsure of the next course of action or if software will survive another 5 years and how my career will look like in the future. Seems like I am engaged in the ice trade and they are about to invent the refrigerator.

Re: Hamilton-Jacobi-Bellman Equation: Reinforcement Learning and Diffusion Models

#17

Post by rsp1984 »

lain98 wrote: post #16

I find myself completely outclassed by mathematicians in my own field. I tried to learn a little math on the side after my regular software engineer gig but I'm completely outclassed by phd's. I am unsure of the next course of action or if software will survive another 5 years and how my career will look like in the future. Seems like I am engaged in the ice trade and they are about to invent the refrigerator.

Don't despair. The key to becoming proficient in advanced subjects like this one is to first try to understand the fundamentals in plain language and pictures in your mind. Ignore the equations. Ask AI to explain the topic at hand at the most fundamental level.

Once the fundamental concepts are understood, what problem is being solved and where the key difficulties are, only then the equations will start to make sense. If you start out with the math, you're making your life unnecessarily hard.

Also, not universally true but directionally true as a rule of thumb, the more equations a text contains the less likely it is that the author itself has truly grasped the subject. People who really grasp a subject can usually explain it well in plain language.

Re: Hamilton-Jacobi-Bellman Equation: Reinforcement Learning and Diffusion Models

#18

Post by magicalhippo »

sfpotter wrote: post #15
measurablefunc wrote: post #2

It's not clear or obvious why continuous semantics should be applicable on a digital computer. This might seem like nitpicking but it's not, there is a fundamental issue that is always swept under the rug in these kinds of analysis which is about reconciling finitary arithmetic over bit strings & the analytical equations which only work w/ infinite precision over the real or complex numbers as they are usually define…

This is what the field of numerical analysis exists for. These details definitely have been treated, but this was done mainly early in the field's history; for example, by people like Wilkinson and Kahan...

I just took some basic numerical courses at uni, but every time we discretized a problem with the aim to implement it on a computer, we had to show what the discretization error would lead to, eg numerical dispersion[1] etc, and do stability analysis and such, eg ensure CFL[2] condition held.

So I guess one might want to do a similar exercise to deriving numerical dispersion for example in order to see just how discretizing the diffusion process affects it and the relation to optimal control theory.

[1]: https://en.wikipedia.org/wiki/Numerical_dispersion

[2]: https://en.wikipedia.org/wiki/Courant%E2%80%93Friedrichs%E2%...

Re: Hamilton-Jacobi-Bellman Equation: Reinforcement Learning and Diffusion Models

#19

Post by imtringued »

measurablefunc wrote: post #2

It's not clear or obvious why continuous semantics should be applicable on a digital computer. This might seem like nitpicking but it's not, there is a fundamental issue that is always swept under the rug in these kinds of analysis which is about reconciling finitary arithmetic over bit strings & the analytical equations which only work w/ infinite precision over the real or complex numbers as they are usually define…

I can't tell if this a troll attempt or not.

If your definition of "algorithm" is "list of instructions", then there is nothing surprising. It's very obvious. The "algorithm" isn't perfect, but a mapping with an error exists.

If your definition of "algorithm" is "error free equivalent of the equations", then the analytical equations do not map to "algorithms". "Algorithms" do not exist.

I mean, your objection is kind of like questioning how a construction material could hold up a building when it is inevitably bound to decay and therefore result in structural collapse. Is it actually holding the entire time or is it slowly collapsing the entire time?

Re: Hamilton-Jacobi-Bellman Equation: Reinforcement Learning and Diffusion Models

#20

Post by numbers_guy »

lain98 wrote: post #16

I find myself completely outclassed by mathematicians in my own field. I tried to learn a little math on the side after my regular software engineer gig but I'm completely outclassed by phd's. I am unsure of the next course of action or if software will survive another 5 years and how my career will look like in the future. Seems like I am engaged in the ice trade and they are about to invent the refrigerator.

I guess I have the opposite experience. I have a post-graduate level of mathematical education and I am dismayed at how little there is to be gained from it, when it comes to AI/ML. Diffusion Models and Geometric Deep Learning are the only two fields where there's any math at all. Many math grads are struggling to find a job at all. They aren't outclassing programmers with their leet math skillz.
Post reply on HN