Automatic Differentiation in 38 lines of Haskell

This is an interesting approach. Haskell is not a symbolic language, but you take advantage of the abstractness of type parameters in function definitions to thread your implementation of "D x" through, and pattern match on that.

It's a neat design pattern. I bet it'd work in Julia too.

Re: Automatic Differentiation in 38 lines of Haskell

#5

Post by ayjtyjtyjgfjc » Sat, Sep 17, 2022, 9:48 PM UTC

green_on_black wrote: post #3

Curious: I don't see the `^` op defined, or is it translated inti `exp` im guessing?

It is one of three exponentiation operators in standard Haskell, no translation involved. https://stackoverflow.com/a/6400630/14768587

Re: Automatic Differentiation in 38 lines of Haskell

#6

Post by sterlind » Sat, Sep 17, 2022, 9:48 PM UTC

mountainriver wrote: post #2

Being purely functional makes this quite easy, still a beauty to see

it's not just the functionalness. you couldn't do it this way in Lisp/Scheme (I think?) because of the lack of multiple dispatch.

If you did (f 'x) for instance, you'd end up with things like (* 2 'x) which would blow up, since Lisp would try to compute the answer instead giving you '(* 2 x) back.

Re: Automatic Differentiation in 38 lines of Haskell

#7

Post by ttesmer » Sat, Sep 17, 2022, 9:52 PM UTC

green_on_black wrote: post #3

Curious: I don't see the `^` op defined, or is it translated inti `exp` im guessing?

Since the type was made an instance of the Num typeclass, any function that can be used with Num's, can now be used on the type (Dual d). As per the Prelude[1], ^ is part of the Num typeclass. Same thing for * for Floating[2]. The hyperbolic tangent can also be used without being explicitly coded, as it can be derived using cosh and sinh!

EDIT: As for the differentiation, it works for ^ since it is just multiplication (https://hackage.haskell.org/package/base-4.17.0.0/docs/src/G...) for which the derivative was defined using the product rule.

[1]: https://hackage.haskell.org/package/base-4.17.0.0/docs/Prelu... [2]: https://hackage.haskell.org/package/base-4.17.0.0/docs/Prelu...

Re: Automatic Differentiation in 38 lines of Haskell

#8

Post by cryptonector » Sat, Sep 17, 2022, 9:59 PM UTC

sterlind wrote: post #6

mountainriver wrote: post #2

Being purely functional makes this quite easy, still a beauty to see

it's not just the functionalness. you couldn't do it this way in Lisp/Scheme (I think?) because of the lack of multiple dispatch. If you did (f 'x) for instance, you'd end up with things like (* 2 'x) which would blow up, since Lisp would try to compute the answer instead giving you '(* 2 x) back.

The Common Lisp Object System has multiple dispatch.

Re: Automatic Differentiation in 38 lines of Haskell

#9

Post by tikhonj » Sat, Sep 17, 2022, 10:12 PM UTC

green_on_black wrote: post #3

Curious: I don't see the `^` op defined, or is it translated inti `exp` im guessing?

The ^ operator is defined in Haskell's standard library for raising values of any numeric type to non-negative, integral powers. Conceptually a ^ n just expands to a * a * ... * a, but the actual code is a bit more complex[1] for performance reasons.

The neat thing with this approach is that ^ works for any numeric type, including user-defined types like Dual in this example. Since the Dual type can handle calculating derivatives for *, it gets derivatives for ^ for free.

[1]: https://hackage.haskell.org/package/base-4.17.0.0/docs/src/G...

Re: Automatic Differentiation in 38 lines of Haskell

#10

Post by xiphias2 » Sat, Sep 17, 2022, 10:35 PM UTC

sterlind wrote: post #4

This is an interesting approach. Haskell is not a symbolic language, but you take advantage of the abstractness of type parameters in function definitions to thread your implementation of "D x" through, and pattern match on that. It's a neat design pattern. I bet it'd work in Julia too.

Yes, but Julia has both forward and backward differention implemented (backwards it's harder).