Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Composability in Julia: Implementing Deep Equilibrium Models via Neural ODEs

julialang.org

Post reply on HN
Print view View original

31–40 of 53 posts

Re: Composability in Julia: Implementing Deep Equilibrium Models via Neural ODEs

#31

Post by taeric »

gugagore wrote: post #24

> . For example, when we apply convolution filters on images the network consists of repetitive blocks of convolutional layers, and one linear output layer at the very end. It's essentially f(f(f(...f(x))...)) where f is the neural network, and we call this "deep" because of the layers of composition. But what if we make this composition go to infinity? This really does not jive with my understanding. Each layer of,…

I think this works if you consider subscripted fs. That is, it is always the same arity, and the output is the same. So, would be better it they said f_n, where n is the layer if they network.

(I mean this as a question, but don't see an obvious place for a question mark...)

Re: Composability in Julia: Implementing Deep Equilibrium Models via Neural ODEs

#32

Post by orbifold »

jstx1 wrote: post #8

What are some good references on neural ODEs that don't come from the Julia community? I'm looking for theory and applications - when are they good and who is using them for what? I'm asking for sources outside of Julia because I find the coupling of algorithm types to tools kind of strange and the whole SciML trend is kind of opaque to me. (Are people applying ML as a solution to newer problems? Are they using new a…

Neural ODEs are essentially a rebranding of adjoint sensitivity analysis, which has been around in various forms in established solver suites, such as Sundials, PETSC, etc. The machine learning community got a hold of it, cited one book and otherwise happily reinvented everything.

Re: Composability in Julia: Implementing Deep Equilibrium Models via Neural ODEs

#33

Post by Tarrosion »

uoaei wrote: post #25
Tarrosion wrote: post #21

I've been part of and/or following the Julia community since 2015, and Julia is my favorite programming language by a wide margin. Seems like every two months there's a new blog post, usually with some of these folks as authors, that describes...something...to do with ODEs, machine learning, neural ODEs, GPUs, adjoints, scientific machine learning, ... I have never once been able to follow one of these blog posts. Se…

> but still— I'm a long-term Julia fan, professional data scientist, mathy PhD, would hope that's at least table stakes. I don't understand this part. Are you saying you're a "professional data scientist" with a "mathy PhD" who has a "light-to-moderate amount" of machine learning knowledge? How did you get the job? I would expect anyone with a mathy PhD to understand ODEs and PDEs, and neural ODEs are commonly unders…

> Are you saying you're a "professional data scientist" with a "mathy PhD" who has a "light-to-moderate amount" of machine learning knowledge? How did you get the job?

Indeed! My job involves approximately zero machine learning, at least for a narrow or stereotypical definition of machine learning. I work on optimization, domain-specific models inspired by queueing theory, various phsyically-motivated structural models, writing production code for data-heavy products, testing hypotheses in data, brainstorming how existing data can be used to solve new customer problems, writing documentation, communicating with customers, etc.

If you think this isn't data science but know of better nomenclature: please share! I've struggled to write a great job posting for this kind of work, and surely improved nomenclature would help :)

Re: Composability in Julia: Implementing Deep Equilibrium Models via Neural ODEs

#34

Post by diskzero »

uoaei wrote: post #28
diskzero wrote: post #26

Earlier quoted context omitted.

Have you been in interview loops or worked with bread and butter data scientists performing common tasks? I am curious what your view of what most data scientists do day in and day out?

What about tasks being common makes a light-to-moderate understanding of machine learning sufficient? Processes initiated by data scientists during the execution of their role will tend to fail silently. What is meant here is, throwing an inappropriate model at otherwise good data produces unreliable (catastrophic in certain situations) results, but produces results nonetheless. Without the proper discernment of the…

This is helpful, and I agree with you. There are some "data scientists" who are able to use PyTorch, TensorFlow, etc. and modify code in a Jupyter Notebook without knowing the larger ramifications of the work they are doing.

Re: Composability in Julia: Implementing Deep Equilibrium Models via Neural ODEs

#36

Post by jimsimmons »

Tarrosion wrote: post #21

I've been part of and/or following the Julia community since 2015, and Julia is my favorite programming language by a wide margin. Seems like every two months there's a new blog post, usually with some of these folks as authors, that describes...something...to do with ODEs, machine learning, neural ODEs, GPUs, adjoints, scientific machine learning, ... I have never once been able to follow one of these blog posts. Se…

I consider myself to be reasonably good at ML and work for one of the cool labs and this stuff is impenetrable to me. Stuff like this are deceptively written and a cynic might say that they are primarily to show off rather than help.

Re: Composability in Julia: Implementing Deep Equilibrium Models via Neural ODEs

#38

Post by ViralBShah »

Tarrosion wrote: post #21

I've been part of and/or following the Julia community since 2015, and Julia is my favorite programming language by a wide margin. Seems like every two months there's a new blog post, usually with some of these folks as authors, that describes...something...to do with ODEs, machine learning, neural ODEs, GPUs, adjoints, scientific machine learning, ... I have never once been able to follow one of these blog posts. Se…

The SciML tutorials are a good start: https://github.com/SciML/SciMLTutorials.jl

And also the 18.337 lecture notes (probably also has videos available): https://github.com/mitmath/18337

Re: Composability in Julia: Implementing Deep Equilibrium Models via Neural ODEs

#40

Post by amkkma »

Tarrosion wrote: post #33
uoaei wrote: post #25

Earlier quoted context omitted.

> but still— I'm a long-term Julia fan, professional data scientist, mathy PhD, would hope that's at least table stakes. I don't understand this part. Are you saying you're a "professional data scientist" with a "mathy PhD" who has a "light-to-moderate amount" of machine learning knowledge? How did you get the job? I would expect anyone with a mathy PhD to understand ODEs and PDEs, and neural ODEs are commonly unders…

> Are you saying you're a "professional data scientist" with a "mathy PhD" who has a "light-to-moderate amount" of machine learning knowledge? How did you get the job? Indeed! My job involves approximately zero machine learning, at least for a narrow or stereotypical definition of machine learning. I work on optimization, domain-specific models inspired by queueing theory, various phsyically-motivated structural mode…

in what is your phd?
Post reply on HN