This is the first time i've read about differentiable programming and it was incredibly interesting. Also, it's the first time I've came across this blog and it's wonderful. The minimal interactions really helped me visualise the content and it helped with comprehension. Great job.
Differentiable programming from scratch
41–50 of 112 posts
Re: Differentiable programming from scratch
#42This is the first time i've read about differentiable programming and it was incredibly interesting. Also, it's the first time I've came across this blog and it's wonderful. The minimal interactions really helped me visualise the content and it helped with comprehension. Great job.
Yes, very beautiful stuff. Only thing I don't like: The text in the SVG's is not texed, but that's difficult to achieve, I guess.
Re: Differentiable programming from scratch
#43Earlier quoted context omitted.
Two points, (1) everything which makes programs useful is impure device access and state change, discretely sequenced over time (2) grad. desc. et al. do not learn discrete constraints (hence why NNs are bad at learning operators: they cant. x+x is defined fa. x; not fa x. in the training set).
> everything which makes programs useful is impure device access and state change, discretely sequenced over time I haven't heard about this before actually. I'd love to hear more about this! "Impure," here, is PL terminology for functions that affect global state/arguments when you run them. right? So, brainstorming a bit, what this means is that making a diff. programming language that treats a NN module as a pure…
(1) Yes, the modern ML/AI lot seem to ambiguously use a purely mathematical meaning to "computer" -- which is useless. As useless as any pure mathematics. If we only had this a "computer" would be a theoretical curiosity, like a 200-dim sphere.
The real-world computers we care about run algorithms whose semantics is given by the properties of the devices real computers use. This double meaning to "computer" has caused a lot of superstition in the ML/AI space.
Real computers are engineering devices which shuffle electrical signals around to useful devices.
There is no reason to think that "pure algorithms" have any use at all, as with, eg., a 200-dim sphere. They're only useful if they can be given a semantics which exploits useful properties of devices. (cf. with physics, where a 200-dim sphere could be useful if it models some actual system).
(2) This isn't enough. Consider learning the rules of chess; or likewise, the inference rules of mathematics. f(x) = 2x^2, f'(x) = 4x, etc.
Search spaces constructed for a grad. desc. search are very infinite; and the solutions we need are infinitely precise. Discrete approaches to search(ing for solutions) are necessary.
Re: Differentiable programming from scratch
#44Earlier quoted context omitted.
Two points, (1) everything which makes programs useful is impure device access and state change, discretely sequenced over time (2) grad. desc. et al. do not learn discrete constraints (hence why NNs are bad at learning operators: they cant. x+x is defined fa. x; not fa x. in the training set).
Two counter-points (appreciate some counter-counter-points :): 1) The discrete sequencing is an epiphenomenon. The underlying processes are continuous changes in voltage and current flows. (I'm not sure if Planck scale considerations can throw a wrench in this though. Would love to be educated here.) 2) Our brains do not have ostensibly discrete neural processors. I don't think gradient descent is comparable to how t…
You're making a genetic fallacy here: that since the origin/ground of something has property C, it's product must have it too. This isnt so.
I agree that reality is fundamentally continuous. However cognition isnt; and many things arent.
A frequency is discrete. A length is continuous. These properties aren't eliminable for one another.
Here, whilst i'd agree that all physical process going on (everywhere) have essential continuous properties; they also have essential discrete ones. The issue is that grad. desc. alone does not give you the right kind of discrete ones.
Re: Differentiable programming from scratch
#45Nice! I asked something about this a while ago: https://news.ycombinator.com/item?id=31044118
Thanks for the link. Interesting conversations and anecdotes.
Don’t worry about the fact that you don’t understand anything. Just keep trying to get results. If you focus on getting results, the understanding will come to you naturally. It’s how I learned ML.
Watching presentations is nice, but tinkering with working code is so much nicer. It’s no surprise you came away feeling like you don’t actually know what you thought you knew — the only way to understand it is to go get some experience.
Also don’t worry that it takes a long time to understand something. This stuff is inherently hard. (My answer to your post is “Yes, of course I feel that way too! I don’t understand a damn thing in most presentations till I go code it myself.” So don’t feel alone!)
Re: Differentiable programming from scratch
#46One thing that I've thought about, when using calculus in programs, you're often dealing with a very small (but finite) Δx, rather than an infinitesimal 𝛿x. And I don't think that the common differential equation is the ideal form when dealing with that situation. (Take for example g(x) = -f(-x), the gradients at x and -x are not equivalent for Δx, but would be for 𝛿x.) Anyway, (limit(h -> 0)((f(x + h) - f(x))/h) W…
Is h use instead of 𝛿 or Δ, because it is applicable to both partial and normal differential. 𝛿 Is always a partial symbol to me.
d in dx whilst is not the same as h, as it stands its own way now as “operator”.
Re: Differentiable programming from scratch
#47You don't need induction for (x+x'e)^n+1. The binomial formula can be applied once the arithmetic on dual numbers is introduced. > We can use this result to prove the same property for any smooth function f. Examining the Taylor expansion of f at zero (also known as its Maclaurin series): That's not exactly true. For example, arctan = tan^-1 is smooth on R, but its Taylor series only converges for |x| 1, where simila…
> That's not exactly true. For example, arctan = tan^-1 is smooth on R, but its Taylor series only converges for |x|This issue creeps in a lot but I think it comes from the complex analysis, where differentiable function is always smooth and analytic (holomorphic).
Re: Differentiable programming from scratch
#48One thing that I've thought about, when using calculus in programs, you're often dealing with a very small (but finite) Δx, rather than an infinitesimal 𝛿x. And I don't think that the common differential equation is the ideal form when dealing with that situation. (Take for example g(x) = -f(-x), the gradients at x and -x are not equivalent for Δx, but would be for 𝛿x.) Anyway, (limit(h -> 0)((f(x + h) - f(x))/h) W…
If you're interested in how to best calculate derivatives numerically using a finite difference rather than an infinitesimal, try having a look at the Wikipedia page on numerical differentiation. The punchline is that you can control the error in f'(x) much better if you use many displacements, ie. f'(x) = a1 f(x + b1 h) + a2 f(x + b2 h) + a3 f(x + b3 h)..., where the bi are chosen and the ai depend on the bi and h.
Re: Differentiable programming from scratch
#49One thing that I've thought about, when using calculus in programs, you're often dealing with a very small (but finite) Δx, rather than an infinitesimal 𝛿x. And I don't think that the common differential equation is the ideal form when dealing with that situation. (Take for example g(x) = -f(-x), the gradients at x and -x are not equivalent for Δx, but would be for 𝛿x.) Anyway, (limit(h -> 0)((f(x + h) - f(x))/h) W…
You really just derived the central derivative. Yes, this derivative is more accurate. The issue is that you can't use it on the edges. For the boundaries you're stuck with lower precision.
Here's a resource you might be interested in http://www2.math.umd.edu/~dlevy/classes/amsc466/lecture-note...
Re: Differentiable programming from scratch
#50One thing that I've thought about, when using calculus in programs, you're often dealing with a very small (but finite) Δx, rather than an infinitesimal 𝛿x. And I don't think that the common differential equation is the ideal form when dealing with that situation. (Take for example g(x) = -f(-x), the gradients at x and -x are not equivalent for Δx, but would be for 𝛿x.) Anyway, (limit(h -> 0)((f(x + h) - f(x))/h) W…
An entire subfield of analysis called Stenciling exists, for this purpose. Depending on the function or differential equation / system, different stencils are used.
Stenciling doesn't just deal with the derivative but it tries to come up with approximations involving a fixed number of sample points, the stencil, for any differential operator, ie. the Laplacian, higher order derivatives, etc.
> The term "stencil" was coined for such patterns to reflect the concept of laying out a stencil in the usual sense over a computational grid to reveal just the numbers needed at a particular step. [2]
[0] https://en.wikipedia.org/wiki/Five-point_stencil
[1] https://en.m.wikipedia.org/wiki/Finite_difference_coefficien...
[2] https://en.wikipedia.org/wiki/Stencil_(numerical_analysis)