Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Differentiable programming from scratch

thenumb.at

Post reply on HN
Print view View original

31–40 of 112 posts

Re: Differentiable programming from scratch

#31

Post by mjburgess »

muds wrote: post #30

I find differentiable programming languages really fascinating. Think about this: a differentiable programming language is still a programming language. If the language is designed to facilitate a smooth optimization landscape, it's actually possible to "learn" programs with gradient descent. This opens the door to a lot of cool possibilities: - programming languages which use neural networks as primitive functions (…

Two points,

(1) everything which makes programs useful is impure device access and state change, discretely sequenced over time

(2) grad. desc. et al. do not learn discrete constraints (hence why NNs are bad at learning operators: they cant. x+x is defined fa. x; not fa x. in the training set).

Re: Differentiable programming from scratch

#32

Post by kragen »

data_maan wrote: post #28
kragen wrote: post #23

Earlier quoted context omitted.

Agreed about the binomial formula, but don't you need induction to prove the binomial theorem? That is, if you have some sort of set that's like the naturals except that Peano's axiom of induction doesn't apply (such as the naturals plus Alberto and Cristina, who are one another's successors) is the binomial theorem necessarily true in it? Agreed about Maclaurin series. I didn't know there existed smooth functions wi…

Rather then repeating various arguments, I think it's best to just mention this link, which provides explanations and illuminating examples of the phenomenon of smooth functions with diverging Taylor series: https://mathoverflow.net/questions/72/whats-an-example-of-a-... Quote by Pietro Majer from the link: "it is more useful to communicate the (historically non-trivial) idea that there is a difference between a func…

Thanks! I wonder why the example of a smooth function with a diverging Taylor series here is so much more complicated than arctan.

Re: Differentiable programming from scratch

#33

Post by [deleted] »

selestify wrote: post #22
radarsat1 wrote: post #18

Earlier quoted context omitted.

> So, in case of (stochastic) gradient descent to train a neural network, that would be the neural network that you want to use AD on, not the gradient descent algorithm. Obviously. The specific proposal of differential programming as a paradigm that goes beyond simple applications of gradient descent for optimising NNs is exactly to apply gradient descent to optimise learning (and other) algorithms themselves. This…

Wow, it’s interesting that a paper from 2016 is already classic. This field moves fast!

[deleted]

Re: Differentiable programming from scratch

#34

Post by muds »

mjburgess wrote: post #31
muds wrote: post #30

I find differentiable programming languages really fascinating. Think about this: a differentiable programming language is still a programming language. If the language is designed to facilitate a smooth optimization landscape, it's actually possible to "learn" programs with gradient descent. This opens the door to a lot of cool possibilities: - programming languages which use neural networks as primitive functions (…

Two points, (1) everything which makes programs useful is impure device access and state change, discretely sequenced over time (2) grad. desc. et al. do not learn discrete constraints (hence why NNs are bad at learning operators: they cant. x+x is defined fa. x; not fa x. in the training set).

> everything which makes programs useful is impure device access and state change, discretely sequenced over time

I haven't heard about this before actually. I'd love to hear more about this! "Impure," here, is PL terminology for functions that affect global state/arguments when you run them. right? So, brainstorming a bit, what this means is that making a diff. programming language that treats a NN module as a pure function won't actually be beneficial? I'm not sure if I'm drawing the correct conclusion but this is a really interesting point. Don't have an answer for this (yet!).

> grad. desc. et al. do not learn discrete constraints

Great Point! To push back a little on this. You're right that any discrete constraint will always mess up the smoothness of the function (eg: less-than-g is not smooth at x=g). However, we can engineer our way around this by relaxing a discrete constraint to its closest smooth approximation! So, we can implement the less-than-g function as a sigmoid that is shifted by +/-g. This introduces a parameter to control the slope of the sigmoid. In practice, I haven't had much difficulty learning programs even with a really steep slope for the sigmoid.

Re: Differentiable programming from scratch

#35

Post by infogulch »

Related post:

Differentiable Programming – A Simple Introduction | 159 points, 3 months ago, 49 comments | https://news.ycombinator.com/item?id=31000709

There I mention "The simple essence of automatic differentiation" which posits that by keeping the a differentiated function paired together with its integrand then many functional transformations -- including integration, subexpression deduplication, and even automatic conversion from forward mode to reverse mode -- would be greatly simplified. Allegedly this claim is not backed up by a practical demonstration yet, but imo this is a very intriguing approach.

Re: Differentiable programming from scratch

#36

Post by radarsat1 »

selestify wrote: post #22
radarsat1 wrote: post #18

Earlier quoted context omitted.

> So, in case of (stochastic) gradient descent to train a neural network, that would be the neural network that you want to use AD on, not the gradient descent algorithm. Obviously. The specific proposal of differential programming as a paradigm that goes beyond simple applications of gradient descent for optimising NNs is exactly to apply gradient descent to optimise learning (and other) algorithms themselves. This…

Wow, it’s interesting that a paper from 2016 is already classic. This field moves fast!

Well, I'd say it's a "classic" within the field of differentiable programming, which is a new field, so... everything is relative.

Re: Differentiable programming from scratch

#37

Post by yarg »

One thing that I've thought about, when using calculus in programs, you're often dealing with a very small (but finite) Δx, rather than an infinitesimal 𝛿x.

And I don't think that the common differential equation is the ideal form when dealing with that situation.

(Take for example g(x) = -f(-x), the gradients at x and -x are not equivalent for Δx, but would be for 𝛿x.)

Anyway,

    (limit(h -> 0)((f(x + h) - f(x))/h)
Why do we even need h? It's just (the infinitesimal) 𝛿x multiplied by an arbitrary finite constant.

(You're dealing with a linear equivalent infinitesimal subsection of the graph - so 𝛿x and 𝛿y scale linearly with each other.)

So remove h:

    ((f(x + 𝛿x) - f(x))/𝛿x)
Looks cleaner.

    (f(x + 𝛿x) - f(x))/𝛿x ≡ (f(x) - f(x - 𝛿x))/𝛿x ≡ (f(x + 𝛿x) - f(x - 𝛿x))/(2 * 𝛿x)
You can take the gradient from x to (x + 𝛿x) or from (x - 𝛿x) to x, or from (x - 𝛿x) to (x + 𝛿x).

(The cube-root of the other three forms is also a valid differential equation, if you want to be evil about it.)

    (f(x + 𝛿x) - f(x - 𝛿x))/(2 * 𝛿x)
(That's a nicer form for programmatic use; resolving the earlier mentioned gradient issue when dealing with finite deltas.)

Another thing I noticed: the standard (h -> 0) form eliminates all parts of the gradient containing 𝛿x values - which is fine for infinitesimal 𝛿x, but is less ideal for finite Δx.

Re: Differentiable programming from scratch

#38

Post by usgroup »

mjburgess wrote: post #31
muds wrote: post #30

I find differentiable programming languages really fascinating. Think about this: a differentiable programming language is still a programming language. If the language is designed to facilitate a smooth optimization landscape, it's actually possible to "learn" programs with gradient descent. This opens the door to a lot of cool possibilities: - programming languages which use neural networks as primitive functions (…

Two points, (1) everything which makes programs useful is impure device access and state change, discretely sequenced over time (2) grad. desc. et al. do not learn discrete constraints (hence why NNs are bad at learning operators: they cant. x+x is defined fa. x; not fa x. in the training set).

I think this is partially true; there is some support for logical statements and control flow in differentiable programs -- at least in Jax. Further, think Deep Mind have a recent paper on a DL sequence learning methodology able to learn control strategies for lots of games simultaneously. I think this is a good example of learning discrete constraints with an NN.

Re: Differentiable programming from scratch

#39

Post by Sirenos »

mjburgess wrote: post #31
muds wrote: post #30

I find differentiable programming languages really fascinating. Think about this: a differentiable programming language is still a programming language. If the language is designed to facilitate a smooth optimization landscape, it's actually possible to "learn" programs with gradient descent. This opens the door to a lot of cool possibilities: - programming languages which use neural networks as primitive functions (…

Two points, (1) everything which makes programs useful is impure device access and state change, discretely sequenced over time (2) grad. desc. et al. do not learn discrete constraints (hence why NNs are bad at learning operators: they cant. x+x is defined fa. x; not fa x. in the training set).

Two counter-points (appreciate some counter-counter-points :):

1) The discrete sequencing is an epiphenomenon. The underlying processes are continuous changes in voltage and current flows. (I'm not sure if Planck scale considerations can throw a wrench in this though. Would love to be educated here.)

2) Our brains do not have ostensibly discrete neural processors. I don't think gradient descent is comparable to how the brain learns, but I think there is some reason to think that it is possible to learn symbolic processes in spite of having a processor that isn't especially built for it.

Post reply on HN