Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Differentiable programming from scratch

thenumb.at

Post reply on HN
Print view View original

81–90 of 112 posts

Re: Differentiable programming from scratch

#81

Post by ogogmad »

goldenkey wrote: post #71
ogogmad wrote: post #68

Earlier quoted context omitted.

You've just described the dual numbers, which provide a way of implementing forward-mode autodiff. > If the derivative is abnormal in a way such that the normal > truncated calculative formulas would conceal it, then AD > would fail to give an accurate result as well. What I'm seeing directly contradicts that. I've tested the example above (the one I called f(x,a) with a=1e-36, trying to find its derivative at x=0),…

> Autodiff using the dual numbers has pulled off the seemingly impossible. It's not seemingly impossible, if you understand that the chain rule for f' is just being executed at the same time as the calculation of f by having derivatives for basic operations already defined. However, like I said, if you have a calcuation method that hides derivatives in terms that have been truncated, then this will not save you. (-1)…

> Try a different series or calculation method, and dual numbers will get you a wildly different result

No. The example works because:

  While the exact value of f(x,a) isn't 1, given any inexact representation of real numbers like floating point or fixed point, the value of "a" can be chosen so that f(x,a) has 1 as its closest representation.
Trying to compute f(x,a) differently isn't going to change that, so stencilling methods are never going to work here. But autodiff will always work. This means I win your challenge.

Your other claims are probably gibberish. You need to provide an example.

Re: Differentiable programming from scratch

#82

Post by data_maan »

mjburgess wrote: post #43
muds wrote: post #34

Earlier quoted context omitted.

> everything which makes programs useful is impure device access and state change, discretely sequenced over time I haven't heard about this before actually. I'd love to hear more about this! "Impure," here, is PL terminology for functions that affect global state/arguments when you run them. right? So, brainstorming a bit, what this means is that making a diff. programming language that treats a NN module as a pure…

(1) Yes, the modern ML/AI lot seem to ambiguously use a purely mathematical meaning to "computer" -- which is useless. As useless as any pure mathematics. If we only had this a "computer" would be a theoretical curiosity, like a 200-dim sphere. The real-world computers we care about run algorithms whose semantics is given by the properties of the devices real computers use. This double meaning to "computer" has cause…

> As useless as any pure mathematics

Are you hearing yourself talk? Do you know why you have (to just name one example out of many) thousands of pictures on your phone, and not just a few? Because of pure mathematics. Because of compression: Even JPEG2000 from back in the day uses intricate and beautiful compression algorithms based on wavelets.

Re: Differentiable programming from scratch

#83

Post by data_maan »

godelski wrote: post #49
yarg wrote: post #37

One thing that I've thought about, when using calculus in programs, you're often dealing with a very small (but finite) Δx, rather than an infinitesimal 𝛿x. And I don't think that the common differential equation is the ideal form when dealing with that situation. (Take for example g(x) = -f(-x), the gradients at x and -x are not equivalent for Δx, but would be for 𝛿x.) Anyway, (limit(h -> 0)((f(x + h) - f(x))/h) W…

You really just derived the central derivative. Yes, this derivative is more accurate. The issue is that you can't use it on the edges. For the boundaries you're stuck with lower precision. Here's a resource you might be interested in http://www2.math.umd.edu/~dlevy/classes/amsc466/lecture-note...

This!

I always find it curious how computer scientists end up rediscovering things known for decades, and then get all hyped about it ...

Re: Differentiable programming from scratch

#84

Post by data_maan »

oddity wrote: post #25
data_maan wrote: post #21

Earlier quoted context omitted.

> In a toy example like this where you're doing everything manually then you probably don't care, but for larger systems, it gets tiring to do the mental equivalent of assembly programming. But by this argument (which sounds plausible to me) you have defeated your previous claim that differential programming is really a new paradigm, as it seems you have adopted what bmitc wrote earlier, that differential programming…

There's no contradiction: autodiff is a method of implementing differentiable programming. In this example, it is implemented as a type that handles a trace of a program, but everything else is left to the programmer. This is a problem because most of the code I would want to write is not a single trace! Analogously, I could write a program in C that does message sends and organizes code in a design pattern called "o…

Seems we're arguing about definitions. Currently differentiable programming seems to be this vaguely defined term (I don't get what you mean by smooth substitution), with autodiff being its only (proper) instantiation.

You say autodiff is actually not representative of differentiable programming. But if there aren't any other good examples that illustrate differentiable programming, how is differentiable programming (currently) more than autodiff?...

Re: Differentiable programming from scratch

#85

Post by joe__f »

ogogmad wrote: post #70
joe__f wrote: post #65

Earlier quoted context omitted.

Finite difference derivatives are the basis for a range of numerical methods for solving differential equations. Given that these techniques underpin various areas of modern engineering, I'd say your statement that 'differentiation [...] cannot be accurately estimated using numerical methods' is demonstrably incorrect. I think you're fishing for something like 'there exist functions whose derivatives cannot be accura…

> Finite difference derivatives are the basis for a range of numerical methods for solving differential equations. I thought the basis usually involved rewriting the equation in terms of integrals, which can be estimated using quadrature. Numerical quadrature does not have the same problems as finite differencing because integration is a continuous operator. How accurate are these methods you're proposing? Could they…

Well, there are many different methods and certainly some are based on numerical integration as you say. The most simple method based on finite difference derivatives is Euler's method which uses the scheme given by OP, with error O(h). This can be extended to Runge-Kutta methods which use a finite difference with n terms and have error O(h^(n-1)), and which to my knowledge are common and widely used to solve ODEs. You can read about Runge-Kutta methods on Wikipedia, and they also have a page on numerical methods for ODEs which gives an overview of different methods.

I don't know what autodiff is, or how the dual numbers fit in here. I'd be interested to learn if you want to explain

Re: Differentiable programming from scratch

#86

Post by data_maan »

bmitc wrote: post #75
data_maan wrote: post #29

Earlier quoted context omitted.

> I've been looking into this stuff lately, to try and justify mathematically why automatic differentiation works. What do you mean by that? Isn't the theory (at least the mathematical, abstract version of it) already fully represented in books like Griewank that I mentioned in another comment here in this post? I remember seeing somewhere a rather well develop theory using dual numbers (but this was a purely mathema…

> Isn't the theory (at least the mathematical, abstract version of it) already fully represented in books like Griewank that I mentioned in another comment here in this post? I guess it's in there somewhere, but like many of the papers, I find that book to overcomplicate things and just not to my liking. I also don't see any mention of dual numbers, although it's probably somewhat implicit in the development (maybe).…

Yes, the book is hard to read. I just went through a few chapters of it.

> One of the papers posted led me to be more comfortable with the concept of differentiable programming as a distinct thing.

I'd still be interested in knowing which of the papera that was ;)

Re: Differentiable programming from scratch

#87

Post by ducktective »

sillysaurusx wrote: post #45
ducktective wrote: post #20

Nice! I asked something about this a while ago: https://news.ycombinator.com/item?id=31044118

Thanks for the link. Interesting conversations and anecdotes. Don’t worry about the fact that you don’t understand anything. Just keep trying to get results. If you focus on getting results, the understanding will come to you naturally. It’s how I learned ML. Watching presentations is nice, but tinkering with working code is so much nicer. It’s no surprise you came away feeling like you don’t actually know what you t…

Thanks

Re: Differentiable programming from scratch

#88

Post by data_maan »

bmitc wrote: post #77
ithinkso wrote: post #76

Earlier quoted context omitted.

Yeah, I was only trying to explain where the issue (assuming smooth=analytic, that you rightfully pointed out) might have come from

Ahh, gotcha. I'm trying to figure this out myself. I've always been intrigued by automatic differentiation but have never been satisfied by any article addressing its mathematical justification.

Neither have I ...

Re: Differentiable programming from scratch

#89

Post by goldenkey »

ogogmad wrote: post #81
goldenkey wrote: post #71

Earlier quoted context omitted.

> Autodiff using the dual numbers has pulled off the seemingly impossible. It's not seemingly impossible, if you understand that the chain rule for f' is just being executed at the same time as the calculation of f by having derivatives for basic operations already defined. However, like I said, if you have a calcuation method that hides derivatives in terms that have been truncated, then this will not save you. (-1)…

> Try a different series or calculation method, and dual numbers will get you a wildly different result No. The example works because: While the exact value of f(x,a) isn't 1, given any inexact representation of real numbers like floating point or fixed point, the value of "a" can be chosen so that f(x,a) has 1 as its closest representation. Trying to compute f(x,a) differently isn't going to change that, so stencill…

What don't you understand -- the method of calculating the function is an approximation, thus the AD derivative is dependent on it, and the AD derivative is a derivative of the approximation, not the actual function. Whereas an approximation of the actual derivative is what we are truly after.

> The key take away here is that the map is not the territory. Most nontrivial functions on computers are implemented as some function that that approximates (the map) the mathematical ideal (the territory). Automatic differentiation gives back a completely accurate derivative of the that function (the map) doing the approximation. Furthermore, the accurate derivative of an approximation to the idea (e.g d_my_sin), is less accurate than and approximation to the (ideal) derivative of the ideal (e.g. my_cos). There is no truncation error in the work the AD did; but there is a truncation error in the sense that we are now using a more truncated approximation that we would write ourselves.

https://www.oxinabox.net/2021/02/08/AD-truncation-error.html

AD is great but if you have a calculation method ill-suited for AD, then you'll get shite results. Why is this surprising?

And yeah, stenciling is mostly for PDEs and other state spaces that we don't have a closed form for. It's generally not used for an analytic function. But you can use it for an analytic function if you tailor the stencil to the function. In fact, you'll just yield a truncated Taylor polynomial if you provide a perfect function-specific stencil.

Re: Differentiable programming from scratch

#90

Post by data_maan »

oddity wrote: post #78
data_maan wrote: post #26

Earlier quoted context omitted.

It seems you haven't read my references? As your [2] is my [3] from above! > The existence of a different kind of CPU isn't a meaningful distinction at the level of discussing paradigms. Well, that was my point above: You can't really lump quantum programming together with probabilistic programming, as they are paradigms on different "levels". > practical papers like [2] cite it wistfully as a dream of what could be…

I don't usually respond to old comments, so I don't know if you'll read this, but I hope I can encourage you to think more broadly about what "differentiable programming" means. Different fields have a different perspective on the same set of tools because those tools have different pathological cases in different areas. Context really matters. > Well, that was my point above: You can't really lump quantum programmin…

> don't know if you'll read this

I did read it ;) Because I'm very much interested in this entire topic.

> I hope I can encourage you to think more broadly about what "differentiable programming" means

I'm trying to, but I find it hard. My stance was that differentiable programming seemes like this theory for which only a single example (namely autodiff) existed, as you also said ("autodiff has mostly developed in practical usage, so the use cases are front-running the theory"). But this entire comment of yours really clarified some things for me.

> The main paper I linked [2] is not about autodiff at all. >The quote you cite from [3] is easily misunderstood without that context

You finally convinced me to have a detailed look at this. Thank you for providing the context.

> Conditionals and loops are possible in [2] since it allows church numerals and fixed point combinators but it introduces a nondeterministic sum [...] and is difficult to operationalize. That's what I meant by "wildly uncomputable".

I think I may have misunderstood some of your previous comments (and perhaps vice versa) as it now dawns on me that you use a vocabulary than comes (I guess?) from PL theory and is very different from the one I'm used to, as a mathematician versed in analysis. I'll re-read them.

> Daniel Murfet et al [6] have some related work more directly in the context of machine learning.

I'm actually aware of Daniel Murfet but haven't read his work from last years. Did you have a specific paper from him in mind?

Post reply on HN