Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

A Differentiable Programming System to Bridge ML and Scientific Computing

arxiv.org

Post reply on HN
Print view View original

31–40 of 75 posts

Re: A Differentiable Programming System to Bridge ML and Scientific Computing

#31

Post by ta1234567890 »

Can't comment on the technical details/merit of the paper.

However, this is huge. From a conceptual perspective, everything in the physical world can be modeled as relative rates of change, which are basically differential equations. Having the power to more easily build, run and understand these models will help tremendously in advancing the computer/physical interface as well as all its potential applications.

Thank you to the authors for such a huge contribution. Looking forward to all the cool things that will be made possible through this.

Re: A Differentiable Programming System to Bridge ML and Scientific Computing

#32

Post by eigenspace »

GistNoesis wrote: post #26
KenoFischer wrote: post #10

Earlier quoted context omitted.

I've read the corresponding paper, and while I think it's a fun read, I'm not sure it does particularly much to clear up the underlying confusion here. Reverse mode AD on straight line primitives is describabable in about three lines of exposition. The tricky part is how do you handle control flow (equivalently recursion). The description thereof then depends on what your underlying IR data structure is, but describi…

Currently, I'm currently using TF 1.x for most of my AD needs, but encountering the limits so I'm always looking for neater solutions. Conceptually, I enjoy the functional approach where the standard compiler does the work. But I'm still using some graph based approach, where the order of operations are recorded on a "tape", then replayed it in reverse order. The main reason is to be able to manage the memory and com…

Sounds like you’d love julia!

Re: A Differentiable Programming System to Bridge ML and Scientific Computing

#34

Post by shoyer »

tel wrote: post #20
akhilcacharya wrote: post #9

Earlier quoted context omitted.

They built in Python interop so that you can use numpy in TFSwift

Bridging into NumPy isn’t going to be useful unless it covers some AD system too (XLA?).

XLA is a compiler for array code. It doesn't come with AD -- you need a wrapper like TF or JAX (or Swift, I guess) for that.

Re: A Differentiable Programming System to Bridge ML and Scientific Computing

#35

Post by ddragon »

briantakita wrote: post #27

Interesting project. I question the implications of using ML data shaping to define models. A sufficiently optimized model may be missing fundamental inputs & constraints, yet still perform better than a less optimized model with the correct inputs & constraints. This would effectively create a local maxima of understanding. It seems, while the predictive capabilities of complex systems may increase in some cases, th…

In this case it might be the exact opposite. Regardless of the structure (as long as it has at least one hidden layer and enough parameters), neural networks are universal approximators, which is why they are effectively black boxes that can learn any model. So using more custom structure does not increase the theoretical capability of the network to find more complex models, but instead you're effectively adding more a priori information (by embedding information of the problem within it's structure), which can heavily restrict the search space and therefore making it somewhat less of a magic black box.

An example might be attention models, which while being more complex than the previous models it gives you more information about what the model does (by allowing you to visualize what inputs the following layer is using).

Re: A Differentiable Programming System to Bridge ML and Scientific Computing

#36

Post by 6thaccount2 »

ChrisRackauckas wrote: post #15
6thaccount2 wrote: post #13

Earlier quoted context omitted.

Is there a general way to do this with many of the libraries in Julia? Any chance ya'll could eventually use a test model and script from me in a benchmark test suite. Ex: I have a large synthetic test model from a university and plan on writing some code that uses a bunch of the sparse matrix and linear solve functionality of the language. This would be on a sparse ~70k row/column square matrix.

At least with DiffEq we keep track of a lot of different benchmarks here: https://github.com/JuliaDiffEq/DiffEqBenchmarks.jl and it's always growing. Some of the best benchmarks have come from users who want us to track performance on their problem. Some other libraries have repos around, but not as formalized. We are getting things automated so that we can start helping others. Your model sounds nice because it can…

Awesome! I wish I could give you a real industry one (even bigger), but a synthetic model should adequately reflect the problem domain and they have been the primary models used by researchers in my field for the past twenty years or so.

Edit: which Julia group should I get into contact with over this?

Re: A Differentiable Programming System to Bridge ML and Scientific Computing

#37

Post by cs702 »

KenoFischer wrote: post #4

Happy that this paper finally made it to arxiv. The biggest reason for writing it was to try and showcase some of the breadth of applications we see for really high quality first class AD support at the language level. There are several communities that need this technology, so it makes sense to try and build one system that can address all of them and share tricks. I'm also hoping this gives people a sense of why ou…

Keno: first of all, let me give a big public thank you to you and your colleagues. (For those here who don't know, Keno is listed as one of the authors of the paper, and works closely with Mike Innes, lead author and also lead developer of Zygote. Mike is also an active member of HN.)

Second, let me bring up what I think is a significant issue. My perception is that most deep learning researchers and practitioners -- that includes me -- tend to iterate very rapidly over new ideas. We want to test ideas in code as quickly as possible, and in practice we will not invest the time and effort necessary to figure out how to write cache-optimized kernels (e.g., for GPUs) every time we might need one. In fact, I'd say the default attitude is that it doesn't even make sense for us to use (what we perceive as slow) automated kernel-writing tools to search for and compile optimized kernels. In practice, it always seems faster and easier (from a developer-time standpoint) to write code that leverages existing, inflexible, prebuilt, handtuned kernels that have already proven to work well, are fast, and are already nicely integrated with frameworks like PyTorch and TensorFlow.

There was a good discussion of this topic in the forums a few weeks ago[a] in response to a recent paper published by some folks at Google in which they make a compelling case that this issue is holding back AI research.[b] As an example, they use capsule networks[c], the implementations of which have proven difficult to optimize (e.g., they copy a lot more data around than is strictly necessary, in order to slice and reshape data in a manner that is compatible with preexisting hardware-accelerator kernels).

What are your thoughts on this? Will Zygote provide any improvements or advantages on this front?

--

[a] https://news.ycombinator.com/item?id=20301619

[b] https://dl.acm.org/citation.cfm?id=3321441

[c] https://arxiv.org/abs/1710.09829

Re: A Differentiable Programming System to Bridge ML and Scientific Computing

#39

Post by ChrisRackauckas »

cs702 wrote: post #37
KenoFischer wrote: post #4

Happy that this paper finally made it to arxiv. The biggest reason for writing it was to try and showcase some of the breadth of applications we see for really high quality first class AD support at the language level. There are several communities that need this technology, so it makes sense to try and build one system that can address all of them and share tricks. I'm also hoping this gives people a sense of why ou…

Keno: first of all, let me give a big public thank you to you and your colleagues. (For those here who don't know, Keno is listed as one of the authors of the paper, and works closely with Mike Innes, lead author and also lead developer of Zygote. Mike is also an active member of HN.) Second, let me bring up what I think is a significant issue. My perception is that most deep learning researchers and practitioners --…

There are people in the Julia Lab working on high level tensor operation languages and compilers. It's a hard problem but one that many are interested in solving with Julia.

Re: A Differentiable Programming System to Bridge ML and Scientific Computing

#40

Post by cs702 »

ChrisRackauckas wrote: post #39
cs702 wrote: post #37

Earlier quoted context omitted.

Keno: first of all, let me give a big public thank you to you and your colleagues. (For those here who don't know, Keno is listed as one of the authors of the paper, and works closely with Mike Innes, lead author and also lead developer of Zygote. Mike is also an active member of HN.) Second, let me bring up what I think is a significant issue. My perception is that most deep learning researchers and practitioners --…

There are people in the Julia Lab working on high level tensor operation languages and compilers. It's a hard problem but one that many are interested in solving with Julia.

Chris: As a user of these tools, I cannot tell you how thankful I am for the work you, Keno, Mike and others do. (For those who don't know, Chris works closely with Mike, Keno, and others in the Julia team.)

I recognize that this is a hard problem.[a]

FWIW, I read or heard (can't remember which) that there are people working with Chris Lattner seeking to use predictive AI (instead of search and heuristics) to address this issue in MLIR. Let me add that my understanding of how that would work is very limited, though.

[a] Only superficially. As you can imagine, I'm dealing at a very different level of abstraction with my own set of problems and frustrations.

Post reply on HN