Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Differentiable programming from scratch

thenumb.at

Post reply on HN
Print view View original

101–110 of 112 posts

Re: Differentiable programming from scratch

#101

Post by joe__f »

ogogmad wrote: post #99
joe__f wrote: post #98

Earlier quoted context omitted.

Ah thankyou, it was so neat and automatic that I missed the punchline. It looks like you'll always calculate f and f' together and it's not possible to calculate only f' though? Seems like this could waste time in some applications? I can see how this is the solution to the problem of calculating the numerical derivative of a known function composed of finite combinations of rational functions and a set of specified…

> I can see how this is the solution to the problem of calculating the numerical derivative of a known function composed of finite combinations of rational functions and a set of specified functions, elementary or special. Yes and no. It's worth taking into account that you can apply it to some functions defined using conditions and loops. For instance, if f is the sqrt function defined using Newton's method, then au…

This was an interesting comment and I learnt something new, thank you. It does feel a little bit like you are proselytising the good news of the dual number autodiff method though, when you don't respond to questions on potential downsides and only add information about positives.

Re: Differentiable programming from scratch

#102

Post by ogogmad »

joe__f wrote: post #101
ogogmad wrote: post #99

Earlier quoted context omitted.

> I can see how this is the solution to the problem of calculating the numerical derivative of a known function composed of finite combinations of rational functions and a set of specified functions, elementary or special. Yes and no. It's worth taking into account that you can apply it to some functions defined using conditions and loops. For instance, if f is the sqrt function defined using Newton's method, then au…

This was an interesting comment and I learnt something new, thank you. It does feel a little bit like you are proselytising the good news of the dual number autodiff method though, when you don't respond to questions on potential downsides and only add information about positives.

I'm more familiar with using differentiation in optimisation, where autodiff seems like the best method. I'm also aware of the fact that the differentiation operator is discontinuous, which makes me sceptical of finite-differencing methods. I'm not familiar with numerically solving differential equations, but the natural connection between Runge-Kutta and finite differencing seems to be an interesting one.

Re: Differentiable programming from scratch

#103

Post by Doxin »

godelski wrote: post #94
Doxin wrote: post #92

Earlier quoted context omitted.

I figure it's because math people aren't great at communication with people outside their domain. Point in case being Wikipedia articles on math being absolute gobledygook to non-math people.

To be fair, I can't get computer scientists to read any math books. But I don't think this is quite odd. The only people I know that go out and read math books are mostly (current or former) mathematicians or physicists (which a random sprinkling of others). I'm not sure it is exactly mathematicians faults that your third grade teacher impressed upon you that math is hard and useless.

I don't think math is hard or useless, thank you very much. However if I'm programming a game and want to project a point onto a line I'd prefer to not get told by math folks that I first need to learn all of linear algebra before they can get their solution explained to me. I'm not stupid, I understand plenty of math, I'm just not steeped in the jargon.

Re: Differentiable programming from scratch

#104

Post by godelski »

Doxin wrote: post #103
godelski wrote: post #94

Earlier quoted context omitted.

To be fair, I can't get computer scientists to read any math books. But I don't think this is quite odd. The only people I know that go out and read math books are mostly (current or former) mathematicians or physicists (which a random sprinkling of others). I'm not sure it is exactly mathematicians faults that your third grade teacher impressed upon you that math is hard and useless.

I don't think math is hard or useless, thank you very much. However if I'm programming a game and want to project a point onto a line I'd prefer to not get told by math folks that I first need to learn all of linear algebra before they can get their solution explained to me. I'm not stupid, I understand plenty of math, I'm just not steeped in the jargon.

I feel like you're responding as if I attacked you. I'm not sure why. Maybe it was the comment about not being able to get CS people to read math books but you do? I don't mean that statement in an absolute way (there's no absolutes) but rather as a general case. Of course there are CS people that read math books. It is just my experience that the majority aren't interested in it and I don't think this is an uncommon experience. If you're different, good for you, but don't take general statements personally. I'm not calling you stupid nor am I saying you don't have an interest in math. You're an individual person.

Re: Differentiable programming from scratch

#105

Post by themenomen »

godelski wrote: post #104
Doxin wrote: post #103

Earlier quoted context omitted.

I don't think math is hard or useless, thank you very much. However if I'm programming a game and want to project a point onto a line I'd prefer to not get told by math folks that I first need to learn all of linear algebra before they can get their solution explained to me. I'm not stupid, I understand plenty of math, I'm just not steeped in the jargon.

I feel like you're responding as if I attacked you. I'm not sure why. Maybe it was the comment about not being able to get CS people to read math books but you do? I don't mean that statement in an absolute way (there's no absolutes) but rather as a general case. Of course there are CS people that read math books. It is just my experience that the majority aren't interested in it and I don't think this is an uncommon…

No post body was provided.

Re: Differentiable programming from scratch

#106

Post by yarg »

goldenkey wrote: post #100
yarg wrote: post #80

Earlier quoted context omitted.

Trying to measure the length of a curve f(x). I turn the curve into a fixed infinitesimal thickness (2 * 𝛿t) 'rope' and measure the area. Then I divide the area by the thickness to get the length. The rope is defined as the area between two curves, both of them manipulations of f(x). The curves are generated by moving each point (in f(x)) a constant infinitesimal distance at a tangent to the function (this gives the…

I'm a brainlet or maybe it's the 10+ hours of work I did today but I just can't grasp it right now. There are a lot of ways to take integrals though, so I wouldn't doubt your method could work. Generally though the holes in a lot of these integration methods is their inability to work with functions that are pathological: https://en.wikipedia.org/wiki/Pathological_(mathematics) |group1 = Types of integrals |list1 = *…

(No idea if you'll ever see this, I also have no damned idea what I'm talking about.)

Looking at the Weierstrass function, it's a chaotically convergent series with a non-convergent gradient series.

(The standard differential forms don't apply, because the assumption of linearisation at the scale of 𝛿x is invalid, due to the scale invariant properties of the fractal.)

In that case the integral function would be even more convergent since the 'noise' (pretty much) adds to zero.

So can you differentiate via integration, and get a gradient for a related function (with less noise)?

I messed around with this, taking the areas of a pair of little triangle approximations and adding them to get a ~rectangle.

Then divide that area by 𝛿x to get 𝛿y, and divide that by 𝛿x to get 𝛿y/𝛿x.

This was the form that I reached:

    let y = f(x)
    let F(x) = ∫f(x)𝛿x
    𝛿y/𝛿x = (F(x + 𝛿x) + F(x - 𝛿x) - 2 * F(x))/((𝛿x) ^ 2)
At very least, it passed my polynomial sanity check:

    let F(x) = x^6
    𝛿y/𝛿x = 30(x^4) + 30(x^2)(𝛿x^2) + 2(𝛿x^4)

Re: Differentiable programming from scratch

#107

Post by goldenkey »

yarg wrote: post #106
goldenkey wrote: post #100

Earlier quoted context omitted.

I'm a brainlet or maybe it's the 10+ hours of work I did today but I just can't grasp it right now. There are a lot of ways to take integrals though, so I wouldn't doubt your method could work. Generally though the holes in a lot of these integration methods is their inability to work with functions that are pathological: https://en.wikipedia.org/wiki/Pathological_(mathematics) |group1 = Types of integrals |list1 = *…

(No idea if you'll ever see this, I also have no damned idea what I'm talking about.) Looking at the Weierstrass function, it's a chaotically convergent series with a non-convergent gradient series. (The standard differential forms don't apply, because the assumption of linearisation at the scale of 𝛿x is invalid, due to the scale invariant properties of the fractal.) In that case the integral function would be even…

That's awesome. Glad to see your method worked. If you ever have a chance to encounter a serious mathematician, they can probably give you the proper name of it. Or perhaps we'll have to name it the Yarg integral :-)

Re: Differentiable programming from scratch

#108

Post by yarg »

goldenkey wrote: post #107
yarg wrote: post #106

Earlier quoted context omitted.

(No idea if you'll ever see this, I also have no damned idea what I'm talking about.) Looking at the Weierstrass function, it's a chaotically convergent series with a non-convergent gradient series. (The standard differential forms don't apply, because the assumption of linearisation at the scale of 𝛿x is invalid, due to the scale invariant properties of the fractal.) In that case the integral function would be even…

That's awesome. Glad to see your method worked. If you ever have a chance to encounter a serious mathematician, they can probably give you the proper name of it. Or perhaps we'll have to name it the Yarg integral :-)

I've just been thinking of it as differentiation via integration.

It took me until the end to realise that all I'd done was derive a form for the second derivative of F(x).

Re: Differentiable programming from scratch

#109

Post by goldenkey »

yarg wrote: post #108
goldenkey wrote: post #107

Earlier quoted context omitted.

That's awesome. Glad to see your method worked. If you ever have a chance to encounter a serious mathematician, they can probably give you the proper name of it. Or perhaps we'll have to name it the Yarg integral :-)

I've just been thinking of it as differentiation via integration. It took me until the end to realise that all I'd done was derive a form for the second derivative of F(x).

Ah yes, this is why epsilon squared equals 0 in the dual numbers -- to automatically remove all higher order derivatives when they appear. Exponents on derivatives are indicate of their rank/order.

Re: Differentiable programming from scratch

#110

Post by yarg »

goldenkey wrote: post #109
yarg wrote: post #108

Earlier quoted context omitted.

I've just been thinking of it as differentiation via integration. It took me until the end to realise that all I'd done was derive a form for the second derivative of F(x).

Ah yes, this is why epsilon squared equals 0 in the dual numbers -- to automatically remove all higher order derivatives when they appear. Exponents on derivatives are indicate of their rank/order.

It's the second symmetric derivative.

https://en.wikipedia.org/wiki/Symmetric_derivative#The_secon...

https://en.wikipedia.org/wiki/Second_derivative#Limit

> The limit is called the second symmetric derivative. Note that the second symmetric derivative may exist even when the (usual) second derivative does not.

> This limit can be viewed as a continuous version of the second difference for sequences.

So it actually can do what I intended it to do, nothing original - but there was a degree of satisfaction in deriving it myself.

Post reply on HN