Hack the derivative • Hacker News Board

Re: Hack the derivative

#31

Post by replicant » Sat, Oct 03, 2015, 4:58 PM UTC

eriktaubeneck wrote: post #27

jamessb wrote: post #13

The article comments: "In a lot of respects, it’s quite amazing how accurate many calculations can be made with floating numbers, like orbital mechanics and heat equations" I'm not sure it is so surprising. In Nick Trefethen's Numerical Analysis entry in The Princeton Companion to Mathematics , he notes: "Thus, on a computer, the interval [1 , 2] , for example, is approximated by about 10^16 numbers. It is interestin…

That's an interesting point I had not considered. My view (mostly from the math side, my shallow dive into physical systems was just from a PDE class in grad school) is from the view of unstable differential equations where very small changes to initial conditions can cause massive changes to the output of the system. My uninformed instincts would tell me that even with orders of more digits than atoms in a physical…

Another point of view on why it is not striking that we can model heat conduction well. The heat equation is well-posed, which among other things, means that the solution depends continuously of the input data. And we observe this in many physical systems, that small changes in the cause produces small changes in the effect, therefore, we should expect from good mathematical models of reality to display this property.

Now, orbital mechanics do display unstable behaviour. I don't dare to adventure on how people work around this. https://en.wikipedia.org/wiki/Well-posed_problem

Re: Hack the derivative

#32

Post by cheez » Sat, Oct 03, 2015, 5:07 PM UTC

This was so much fun to read.

Re: Hack the derivative

#33

Post by kmill » Sat, Oct 03, 2015, 6:30 PM UTC

I was trying to figure out how this works algebraically, and I think I understand for polynomials (or power series) at least. Basically, for a really small number a, ai has the property that (ai)^2 is effectively zero, so one can think of this as working with the dual numbers themselves (i.e., throwing out the x^2 term and onward in a power series). The focus on analytic functions is then because they are representable by power series at each point. I think this is what lloda is referring to.

Re: Hack the derivative

#34

Post by jordigh » Sat, Oct 03, 2015, 11:47 PM UTC

jjoonathan wrote: post #22

jordigh wrote: post #20

Earlier quoted context omitted.

wtf Automatic differentiation only works for the simplest functions for which you already know what the Taylor series looks like. For those cases, you might as well just hardcode derivative functions and the basic derivative rules (linearity and chain rule). It is not a general-purpose method. For functions that you can't even express by a simple formula, you still have to rely on finite differencing. Don't call "leg…

Can you comment on the space of functions which you know to be complex differentiable but for which you can't analytically evaluate/approximate the real derivative? If so, I look forward to learning about it. If not, you were throwing stones in a glass house.

Sure, a simple example is the derivative of Riemann zeta for Re s There are also functions that you only know by sampling (e.g. ocean temperatures) for which you assume smoothness. You need to pick an interpolation method, but sometimes you do not interpolate beyond the sampling points, because that's just making up numbers. When you're limited by your original sampling step size, you have little recurse but to compute derivatives by some finite differencing scheme.

Re: Hack the derivative

#35

Post by eriktaubeneck » Sat, Oct 03, 2015, 11:48 PM UTC

cheez wrote: post #32

This was so much fun to read.

Thanks! Glad you enjoyed it.

Re: Hack the derivative

#36

Post by jordigh » Sat, Oct 03, 2015, 11:51 PM UTC

lloda wrote: post #23

jordigh wrote: post #20

Earlier quoted context omitted.

wtf Automatic differentiation only works for the simplest functions for which you already know what the Taylor series looks like. For those cases, you might as well just hardcode derivative functions and the basic derivative rules (linearity and chain rule). It is not a general-purpose method. For functions that you can't even express by a simple formula, you still have to rely on finite differencing. Don't call "leg…

The question was about automatic differentiation vs the method in the article (the complex step method) which is essentially a defective variant of automatic differentiation with a spurious numerical parameter. Automatic differentiation works for any composition of those ‘simplest functions’ you mention, which is quite a lot of stuff, including whole programs. Approximation methods have their place, sure. Sometimes t…

I was objecting to the suggestion that the complex finite differencing scheme was worthless ("legacy") and automatic differentiation is all we should ever do from now on.