Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

A Differentiable Programming System to Bridge ML and Scientific Computing

arxiv.org

Post reply on HN
Print view View original

71–75 of 75 posts

Re: A Differentiable Programming System to Bridge ML and Scientific Computing

#71

Post by eggy »

ddragon wrote: post #67
eggy wrote: post #63

I keep orbiting Julia, because of developments like this one. I am an old Lisper/Scheme guy. How does this compare to say R6RS-AD[1] or cl-autodiff[2] or Deriva for Clojure/Java[3]? I realize Julia has femtolisp inside, yet another Lisp bait for me! [1] https://github.com/qobi/R6RS-AD [2] https://github.com/masonium/cl-autodiff [3] https://github.com/lambder/Deriva

Those are based on overloading functions to add the new behaviors (computing the forward diff or building the graph within a special variable for the reverse mode) which in Julia is implemented for example in [1], [2], [3] and [4], while the one in OP is based on source analysis and transformation. And the similarities with Lisp are more than just the parser being written in it, Julia's programming paradigm is based…

Wow, thank you for such an informative reply!

I am going to try this out. One of my pet peeves with Julia is that my main machine is a Windows 10 box, and things like cudanative.jl say they only install on Mac and Linux. I have an old 2013 Linux box (Lenovo T430u laptop).

BTW, do you recommend Julia Pro install or vanilla Julia and build up for more general technical programming, not just ML and DL.

Re: A Differentiable Programming System to Bridge ML and Scientific Computing

#72

Post by ddragon »

eggy wrote: post #71
ddragon wrote: post #67

Earlier quoted context omitted.

Those are based on overloading functions to add the new behaviors (computing the forward diff or building the graph within a special variable for the reverse mode) which in Julia is implemented for example in [1], [2], [3] and [4], while the one in OP is based on source analysis and transformation. And the similarities with Lisp are more than just the parser being written in it, Julia's programming paradigm is based…

Wow, thank you for such an informative reply! I am going to try this out. One of my pet peeves with Julia is that my main machine is a Windows 10 box, and things like cudanative.jl say they only install on Mac and Linux. I have an old 2013 Linux box (Lenovo T430u laptop). BTW, do you recommend Julia Pro install or vanilla Julia and build up for more general technical programming, not just ML and DL.

I prefer vanilla Julia since it has all the packages at the latest released versions and it's the most used so it's easier to get help. CUDANative.jl documentation doesn't mention being mac/linux only (the only mention was in the 2 year old preview release), and if it is maybe you could try another library like ArrayFire.jl.

And more general technical programming is Julia specialty (it was built for that as a high performance interactive language), the DL hype started after the language was first released.

Re: A Differentiable Programming System to Bridge ML and Scientific Computing

#73

Post by billconan »

StefanKarpinski wrote: post #24
user1241320 wrote: post #5

Is this somehow similar to the new Swift Automatic Differentiation feature? https://github.com/tensorflow/swift/blob/master/docs/Automat...

Yes, it's the same idea. But Julia's differentiable programming capabilities are far more advanced and mature than Swift's. As far as I'm aware, Swift still doesn't support differentiating code with control flow (branches or loops), which, needless to say, eliminates pretty much all non-trivial programs. Compare that to the situation in Julia: ∂P works today on arbitrary programs—like the ray tracer and other example…

How do if conditions handled? They are not differentiatable right? So are loops?

Re: A Differentiable Programming System to Bridge ML and Scientific Computing

#74

Post by billconan »

KenoFischer wrote: post #4

Happy that this paper finally made it to arxiv. The biggest reason for writing it was to try and showcase some of the breadth of applications we see for really high quality first class AD support at the language level. There are several communities that need this technology, so it makes sense to try and build one system that can address all of them and share tricks. I'm also hoping this gives people a sense of why ou…

does the AD algorithm support functions with variant input sizes? For example, the input to a function is an array?

Re: A Differentiable Programming System to Bridge ML and Scientific Computing

#75

Post by StefanKarpinski »

billconan wrote: post #73
StefanKarpinski wrote: post #24

Earlier quoted context omitted.

Yes, it's the same idea. But Julia's differentiable programming capabilities are far more advanced and mature than Swift's. As far as I'm aware, Swift still doesn't support differentiating code with control flow (branches or loops), which, needless to say, eliminates pretty much all non-trivial programs. Compare that to the situation in Julia: ∂P works today on arbitrary programs—like the ray tracer and other example…

How do if conditions handled? They are not differentiatable right? So are loops?

There's a whole paper explaining how it all works :)
Post reply on HN