Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Automatic Differentiation in 38 lines of Haskell

gist.github.com

Post reply on HN
Print view View original

51–60 of 60 posts

Re: Automatic Differentiation in 38 lines of Haskell

#51

Post by iamwil »

sriku wrote: post #28
MrUssek wrote: post #26

To be clear this is a forward mode auto diff implementation, not reverse mode, as might be inferred by the reference to the SPJ talk, correct?

Correct. Reverse mode seems much harder to express in Haskell..i was trying to understand AD some time ago and used Haskell for that (before running into Conal's paper). This way of writing forward AD was easy and it was awesome to see type inference and laziness help me with the understanding. At that time, I tried to code up reverse mode AD and failed to do it with comparable simplicity. If I may ... 1. First attem…

Which paper by Conal?

Re: Automatic Differentiation in 38 lines of Haskell

#52

Post by shele »

krastanov wrote: post #27
magicalhippo wrote: post #23

Earlier quoted context omitted.

I still don't get how sin ends up as cos, without any coordination.

Presumably the Ad package has a list of known derivatives. The Sym package now "automatically" uses it, without ever having to have known of it. The "coordination" is that they both use the "symbol" sin to refer to the idea of sine function.

Interesting that is is this way in Haskell in this example: In Julia, there are community/consensus-based processes to what the "idea" of a symbol is (or rather, the idea of a function with its current methods and its future methods defined for new types) and package interoperability often works in a similar way.

Re: Automatic Differentiation in 38 lines of Haskell

#53

Post by leephillips »

shele wrote: post #52
krastanov wrote: post #27

Earlier quoted context omitted.

Presumably the Ad package has a list of known derivatives. The Sym package now "automatically" uses it, without ever having to have known of it. The "coordination" is that they both use the "symbol" sin to refer to the idea of sine function.

Interesting that is is this way in Haskell in this example: In Julia, there are community/consensus-based processes to what the "idea" of a symbol is (or rather, the idea of a function with its current methods and its future methods defined for new types) and package interoperability often works in a similar way.

There is some confusion about what a symbol is in Julia due to bad tutorial information floating around the web. Here is Stefan Karpinski’s masterful explanation of what a Symbol really is:

https://stackoverflow.com/a/23482257

Re: Automatic Differentiation in 38 lines of Haskell

#54

Post by krastanov »

shele wrote: post #52
krastanov wrote: post #27

Earlier quoted context omitted.

Presumably the Ad package has a list of known derivatives. The Sym package now "automatically" uses it, without ever having to have known of it. The "coordination" is that they both use the "symbol" sin to refer to the idea of sine function.

Interesting that is is this way in Haskell in this example: In Julia, there are community/consensus-based processes to what the "idea" of a symbol is (or rather, the idea of a function with its current methods and its future methods defined for new types) and package interoperability often works in a similar way.

As the sibling comment mentioned, this thread started using the word "symbol" in two different ways. There is the Julia/Lisp/Ruby use of the word symbol which is related to representing code in code and to homoiconicity. That is discussed in the sibling stack-overflow link.

Earlier in this thread though "symbol" was used in a less formal, handwavy way as "a named handle to some concept in the language". In that sense, indeed, the way that Julia adds multiple methods to a single named function and supports multiple dispatch permits amazing interoperability (and has little to do with Julia/Lisp/Ruby symbol datastructures).

Re: Automatic Differentiation in 38 lines of Haskell

#55

Post by shele »

krastanov wrote: post #54
shele wrote: post #52

Earlier quoted context omitted.

Interesting that is is this way in Haskell in this example: In Julia, there are community/consensus-based processes to what the "idea" of a symbol is (or rather, the idea of a function with its current methods and its future methods defined for new types) and package interoperability often works in a similar way.

As the sibling comment mentioned, this thread started using the word "symbol" in two different ways. There is the Julia/Lisp/Ruby use of the word symbol which is related to representing code in code and to homoiconicity. That is discussed in the sibling stack-overflow link. Earlier in this thread though "symbol" was used in a less formal, handwavy way as "a named handle to some concept in the language". In that sense…

Right, my point is just: As new methods are added by the owners of new types it requires a common understanding and community consensus how a function should act on new types (i.e. about the "idea" of the function.)

Re: Automatic Differentiation in 38 lines of Haskell

#56

Post by chowells »

sterlind wrote: post #49
bmitc wrote: post #47

Earlier quoted context omitted.

You can do that with symbolic expressions in all the languages I listed.

In Mathematica, the equivalent of a "function call" would be something like f[x,y]. I can write and run that, and it won't give an error if f, x or y aren't defined. Fundamentally, all atoms in the language are terms of symbols. Expressions are broken down by patterns until they're reduced as far as they can be, then the language happily stops and returns how far it got. It's like Lisp, if everything in Lisp were quo…

> Some kind of reflection in Haskell

In Haskell you can only do that kind of manipulation at compile time. At run time, the source code is long gone. And even at compile time, expressions are a very different type than code that can be run. Haskell just isn't anything like a symbolic language.

As pointed out way upthread, it provides lots of hooks for programmers to have nice syntax for entry points to symbolic systems - if someone writes one.

Re: Automatic Differentiation in 38 lines of Haskell

#57

Post by sriku »

iamwil wrote: post #51
sriku wrote: post #28

Earlier quoted context omitted.

Correct. Reverse mode seems much harder to express in Haskell..i was trying to understand AD some time ago and used Haskell for that (before running into Conal's paper). This way of writing forward AD was easy and it was awesome to see type inference and laziness help me with the understanding. At that time, I tried to code up reverse mode AD and failed to do it with comparable simplicity. If I may ... 1. First attem…

Which paper by Conal?

https://arxiv.org/abs/1804.00746

Re: Automatic Differentiation in 38 lines of Haskell

#58

Post by lalaithion »

green_on_black wrote: post #44
tikhonj wrote: post #9

Earlier quoted context omitted.

The ^ operator is defined in Haskell's standard library for raising values of any numeric type to non-negative, integral powers. Conceptually a ^ n just expands to a * a * ... * a, but the actual code is a bit more complex[1] for performance reasons. The neat thing with this approach is that ^ works for any numeric type, including user-defined types like Dual in this example. Since the Dual type can handle calculatin…

Ah, `^` being naturals-only makes more sense in terms of how it could work with no special logic!

Haskell has three exponentiation operators:

    (^) :: (Num a, Integral b) => a -> b -> a
    (^^) :: (Fractional a, Integral b) => a -> b -> a
    (**) :: Floating a => a -> a -> a
The first one has the un-typed requirement that `b` be nonnegative, and basically allows any value with multiplication (implements the Num typeclass) to be raised to a natural power.

The second one allows any value with multiplication and multiplicative inverses (Fractional) to be raised to an integral power.

The third one allows any value which supports exponents, trig functions, and logarithms to be raised to a power with the same type. `*` is in fact one of the functions one must implement to implement the Floating typeclass.

Re: Automatic Differentiation in 38 lines of Haskell

#60

Post by sriku »

iamwil wrote: post #59
sriku wrote: post #57

Earlier quoted context omitted.

https://arxiv.org/abs/1804.00746

Did this end up helping you with forward or reverse auto diff?

I'm yet to give reverse mode a second try, but my attempts at forward mode (linked in parent post) were on my own without the paper's help. There's a talk that Conal gives that makes the paper a bit more accessible - https://www.youtube.com/watch?v=ne99laPUxN4 - but only a bit more. I know it'll take effort to wrap my head around the implementation details ... but the paper certainly helped clarify things I was stumbling around about.
Post reply on HN