Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Composability in Julia: Implementing Deep Equilibrium Models via Neural ODEs

julialang.org

Post reply on HN
Print view View original

11–20 of 53 posts

Re: Composability in Julia: Implementing Deep Equilibrium Models via Neural ODEs

#11

Post by UncleOxidant »

jstx1 wrote: post #8

What are some good references on neural ODEs that don't come from the Julia community? I'm looking for theory and applications - when are they good and who is using them for what? I'm asking for sources outside of Julia because I find the coupling of algorithm types to tools kind of strange and the whole SciML trend is kind of opaque to me. (Are people applying ML as a solution to newer problems? Are they using new a…

There's some info from the Pyhon/Pytorch camp: https://towardsdatascience.com/neural-odes-with-pytorch-ligh...

I suspect neural ODE work was done in Julia earlier because it was easier given some language features and libraries. But there does seem to be some work on neural ODEs in Python/Pytorch.

Re: Composability in Julia: Implementing Deep Equilibrium Models via Neural ODEs

#12

Post by adgjlsfhk1 »

jstx1 wrote: post #8

What are some good references on neural ODEs that don't come from the Julia community? I'm looking for theory and applications - when are they good and who is using them for what? I'm asking for sources outside of Julia because I find the coupling of algorithm types to tools kind of strange and the whole SciML trend is kind of opaque to me. (Are people applying ML as a solution to newer problems? Are they using new a…

What do you mean by "the coupling of algorithm types to tools" (not judging, just curious).

Re: Composability in Julia: Implementing Deep Equilibrium Models via Neural ODEs

#13

Post by ssivark »

jstx1 wrote: post #8

What are some good references on neural ODEs that don't come from the Julia community? I'm looking for theory and applications - when are they good and who is using them for what? I'm asking for sources outside of Julia because I find the coupling of algorithm types to tools kind of strange and the whole SciML trend is kind of opaque to me. (Are people applying ML as a solution to newer problems? Are they using new a…

The original Neural ODEs paper is quite readable, and by now there are loads of blog posts and even a few talks on the subject.

The basic idea is inspired by the “adjoint method” for ODE solving (so you don’t have to hold in memory all the intermediate layer outputs — which is otherwise necessary to compute the backpropagated gradient signal).

Re: Composability in Julia: Implementing Deep Equilibrium Models via Neural ODEs

#14

Post by avikpal1410 »

cs702 wrote: post #10

Very cool. I have only one question: Has anyone successfully applied DEQs to larger-scale cognitive tasks or benchmarks, as opposed to MNIST, which is a tiny trivial task by today's standards? Think ImageNet-1000, COCO, LVIS, WMT language translation, ..., SuperGLUE. There's a long list of datasets and benchmarks that regular boring fixed-depth NNs tackle with remarkable ease these days. Has anyone anywhere applied D…

MDEQ work applies DEQ to some of the large scale benchmarks you mention: https://arxiv.org/pdf/2006.08656.pdf

Re: Composability in Julia: Implementing Deep Equilibrium Models via Neural ODEs

#15

Post by jstx1 »

adgjlsfhk1 wrote: post #12
jstx1 wrote: post #8

What are some good references on neural ODEs that don't come from the Julia community? I'm looking for theory and applications - when are they good and who is using them for what? I'm asking for sources outside of Julia because I find the coupling of algorithm types to tools kind of strange and the whole SciML trend is kind of opaque to me. (Are people applying ML as a solution to newer problems? Are they using new a…

What do you mean by "the coupling of algorithm types to tools" (not judging, just curious).

More bluntly my question is if SciML is that good, why aren’t more people doing it yet? Why is it limited to a small group of Julia developers and packages?

(There are good possible explanations - it could be very new, have only niche applications, Julia is somehow uniquely suited for it etc. I don’t know)

Re: Composability in Julia: Implementing Deep Equilibrium Models via Neural ODEs

#16

Post by cs702 »

avikpal1410 wrote: post #14
cs702 wrote: post #10

Very cool. I have only one question: Has anyone successfully applied DEQs to larger-scale cognitive tasks or benchmarks, as opposed to MNIST, which is a tiny trivial task by today's standards? Think ImageNet-1000, COCO, LVIS, WMT language translation, ..., SuperGLUE. There's a long list of datasets and benchmarks that regular boring fixed-depth NNs tackle with remarkable ease these days. Has anyone anywhere applied D…

MDEQ work applies DEQ to some of the large scale benchmarks you mention: https://arxiv.org/pdf/2006.08656.pdf

Thank you! The results don't look that great (e.g., EfficientNet models achieve greater accuracy on ImageNet-1000 with ~5x fewer parameters), but the works looks interesting and worthwhile. I'll take a look.

Re: Composability in Julia: Implementing Deep Equilibrium Models via Neural ODEs

#17

Post by ChrisRackauckas »

ssivark wrote: post #13
jstx1 wrote: post #8

What are some good references on neural ODEs that don't come from the Julia community? I'm looking for theory and applications - when are they good and who is using them for what? I'm asking for sources outside of Julia because I find the coupling of algorithm types to tools kind of strange and the whole SciML trend is kind of opaque to me. (Are people applying ML as a solution to newer problems? Are they using new a…

The original Neural ODEs paper is quite readable, and by now there are loads of blog posts and even a few talks on the subject. The basic idea is inspired by the “adjoint method” for ODE solving (so you don’t have to hold in memory all the intermediate layer outputs — which is otherwise necessary to compute the backpropagated gradient signal).

Yeah, though with the method described in that paper you do have to be very careful since it has exponential error growth with the Lipchitz constants of the ODE. See https://aip.scitation.org/doi/10.1063/5.0060697 for details. But that is generally the case in numerical analysis: there's always a simple way to do things, and then there's the way that prevents error growths. Both have different pros and cons.

Re: Composability in Julia: Implementing Deep Equilibrium Models via Neural ODEs

#18

Post by krastanov »

jstx1 wrote: post #15
adgjlsfhk1 wrote: post #12

Earlier quoted context omitted.

What do you mean by "the coupling of algorithm types to tools" (not judging, just curious).

More bluntly my question is if SciML is that good, why aren’t more people doing it yet? Why is it limited to a small group of Julia developers and packages? (There are good possible explanations - it could be very new, have only niche applications, Julia is somehow uniquely suited for it etc. I don’t know)

Some minor clarifications: NeuralODEs are not a Julia invention. I am pretty sure the first papers on the topic were using a python package implementing a rather crude ODE solver in torch or tensorflow. Julia just happens to be light years ahead of any other tool when it comes to solving ODEs, while having many high-quality autodifferentiation packages as well, so it feels natural to use it for these problems. But more importantly, SciML is not just for your typical Machine Learning tasks: being able to solve ODEs and have autodiff over them is incredibly empowering for boring old science and engineering, and SciML has become one of the most popular set of libraries when it comes to unwieldy ODEs.

Re: Composability in Julia: Implementing Deep Equilibrium Models via Neural ODEs

#19

Post by ViralBShah »

jstx1 wrote: post #15
adgjlsfhk1 wrote: post #12

Earlier quoted context omitted.

What do you mean by "the coupling of algorithm types to tools" (not judging, just curious).

More bluntly my question is if SciML is that good, why aren’t more people doing it yet? Why is it limited to a small group of Julia developers and packages? (There are good possible explanations - it could be very new, have only niche applications, Julia is somehow uniquely suited for it etc. I don’t know)

What do you mean by "more" people? Perhaps you mean people who know? Anyone who solves a differential equation in Julia is using the SciML ecosystem of packages. The Julia ecosystem is about 1M users, and lots of people in that ecosystem use these tools.

There's over 100 dependent packages: https://juliahub.com/ui/Packages/OrdinaryDiffEq/DlSvy/5.64.1...

Re: Composability in Julia: Implementing Deep Equilibrium Models via Neural ODEs

#20

Post by UncleOxidant »

jstx1 wrote: post #15
adgjlsfhk1 wrote: post #12

Earlier quoted context omitted.

What do you mean by "the coupling of algorithm types to tools" (not judging, just curious).

More bluntly my question is if SciML is that good, why aren’t more people doing it yet? Why is it limited to a small group of Julia developers and packages? (There are good possible explanations - it could be very new, have only niche applications, Julia is somehow uniquely suited for it etc. I don’t know)

> Why is it limited to a small group of Julia developers and packages?

I don't think there are any gatekeepers limiting it's use. Articles like the one highlighted here help to get the word out to more potential users.

Post reply on HN