For those less familiar with Haskell, here's a C# implementation of dual numbers comparable to this:
https://github.com/naasking/AutoDiffSharp/blob/master/AutoDi...
Automatic Differentiation in 38 lines of Haskell
41–50 of 60 posts
Re: Automatic Differentiation in 38 lines of Haskell
#42Earlier quoted context omitted.
The `sin` function comes from this bit at the end of TFA: instance VectorSpace d => Floating (Dual d) where pi = D pi zero exp (D u u') = D (exp u) (scale (exp u) u') log (D u u') = D (log u) (scale (log u) u') --->sin (D u u') = D (sin u) (scale (cos u) u') cos (D u u') = D (cos u) (scale (-sin u) u') sinh (D u u') = D (sinh u) (scale (cosh u) u') cosh (D u u') = D (cosh u) (scale (sinh u) u') and the `sin` function…
Not quite. This subthread is about the extremely short, one-line implementation mentioned here https://news.ycombinator.com/item?id=32882825 (which merges two unrelated modules (autodiff and symbolic) and uses autodiff to implement symbolic differentiation). Your comment is true for the original 38-line implemention of autodiff at the very top of the thread, but not in this subthread. The 38-line implementation is si…
Yes, indeed, I was referring to TFA, and obviously I was confused.
Re: Automatic Differentiation in 38 lines of Haskell
#43It's not as pretty but forward mode AD isn't that many lines even in C
Re: Automatic Differentiation in 38 lines of Haskell
#44Curious: I don't see the `^` op defined, or is it translated inti `exp` im guessing?
The ^ operator is defined in Haskell's standard library for raising values of any numeric type to non-negative, integral powers. Conceptually a ^ n just expands to a * a * ... * a, but the actual code is a bit more complex[1] for performance reasons. The neat thing with this approach is that ^ works for any numeric type, including user-defined types like Dual in this example. Since the Dual type can handle calculatin…
Ah, `^` being naturals-only makes more sense in terms of how it could work with no special logic!
Re: Automatic Differentiation in 38 lines of Haskell
#45This is an interesting approach. Haskell is not a symbolic language, but you take advantage of the abstractness of type parameters in function definitions to thread your implementation of "D x" through, and pattern match on that. It's a neat design pattern. I bet it'd work in Julia too.
> Haskell is not a symbolic language I'm not sure I understand what this means. What is a symbolic language that excludes languages like Haskell, F#, OCaml, etc.?
A symbolic language is something like mathematica. Mathematica does evaluation as term rewriting with rules for how to rewrite all sorts of things - and if it can't find matching rules, it leaves the expression in the same symbolic form as the input.
Re: Automatic Differentiation in 38 lines of Haskell
#46Where's the vector Jacobian matrix?
You don't need to explicitly form a Jacobian matrix to do AD. The key insight is that the matrix is just a representation of a linear function. In forward mode AD you evaluate the linear function the Jacobian represents (i.e. the derivative) at the same time as evaluating the function you are differentiating, usually without ever explicitly building the matrix. The derivative is a local linear approximation of the original function, so it composes the same way. This allows you to compute the parts of it as you compute the parts of the original function, and pass on the intermediate results alongside the intermediate results of the original calculation.
Re: Automatic Differentiation in 38 lines of Haskell
#47Earlier quoted context omitted.
> Haskell is not a symbolic language I'm not sure I understand what this means. What is a symbolic language that excludes languages like Haskell, F#, OCaml, etc.?
A symbolic language is something like mathematica. Mathematica does evaluation as term rewriting with rules for how to rewrite all sorts of things - and if it can't find matching rules, it leaves the expression in the same symbolic form as the input.
You can do that with symbolic expressions in all the languages I listed.
Re: Automatic Differentiation in 38 lines of Haskell
#48Earlier quoted context omitted.
The `sin` function comes from this bit at the end of TFA: instance VectorSpace d => Floating (Dual d) where pi = D pi zero exp (D u u') = D (exp u) (scale (exp u) u') log (D u u') = D (log u) (scale (log u) u') --->sin (D u u') = D (sin u) (scale (cos u) u') cos (D u u') = D (cos u) (scale (-sin u) u') sinh (D u u') = D (sinh u) (scale (cosh u) u') cosh (D u u') = D (cosh u) (scale (sinh u) u') and the `sin` function…
Not quite. This subthread is about the extremely short, one-line implementation mentioned here https://news.ycombinator.com/item?id=32882825 (which merges two unrelated modules (autodiff and symbolic) and uses autodiff to implement symbolic differentiation). Your comment is true for the original 38-line implemention of autodiff at the very top of the thread, but not in this subthread. The 38-line implementation is si…
However, if the symbolic package knows that
cos(a+b) == cos(a)cos(b) - sin(a)sin(b)
then it works.
Something must be known in advance about the relationship between sin and cos for addition, otherwise you cannot go from one to the other (and the basic
cos(a)^2+sin(a)^2=1
is not enough for that.
Re: Automatic Differentiation in 38 lines of Haskell
#49Earlier quoted context omitted.
A symbolic language is something like mathematica. Mathematica does evaluation as term rewriting with rules for how to rewrite all sorts of things - and if it can't find matching rules, it leaves the expression in the same symbolic form as the input.
You can do that with symbolic expressions in all the languages I listed.
In Mathematica, the equivalent of a "function call" would be something like f[x,y]. I can write and run that, and it won't give an error if f, x or y aren't defined. Fundamentally, all atoms in the language are terms of symbols. Expressions are broken down by patterns until they're reduced as far as they can be, then the language happily stops and returns how far it got.
It's like Lisp, if everything in Lisp were quoted, and eval consisted of pattern matching and applying substitution rules.
As an example, say in Haskell I have foo x y = x + 2/y. Can you write a function that takes any function in, and replaces all + with *? in Mathematica you can: foo[x_,y_] := x + 2/y; foo[x,y] /. l_ + r_ -> l*r.
I'm sure you can do this stuff in other languages if you try hard enough. Some kind of reflection in Haskell or F#. In Lisp, grab and quote the definition of foo and apply macro machinery to it. But that's not the tao of those languages, while it is the essence of Mathematica.
Re: Automatic Differentiation in 38 lines of Haskell
#50Earlier quoted context omitted.
it's not just the functionalness. you couldn't do it this way in Lisp/Scheme (I think?) because of the lack of multiple dispatch. If you did (f 'x) for instance, you'd end up with things like (* 2 'x) which would blow up, since Lisp would try to compute the answer instead giving you '(* 2 x) back.
The Common Lisp Object System has multiple dispatch.
But math operations are not generic functions. So you'd need to create your own version of all math functions. And that won't compose with independently developed libraries (unlike the Haskell example elsewhere in the thread).