Composability in Julia: Implementing Deep Equilibrium Models via Neural ODEs

Lots to say here. First of all, the community growth has been pretty tremendous and I couldn't really ask for more. We're seeing tens of thousands of visitors to the documentation of various packages, and have some high profile users. For example, NASA showing a 15,000x acceleration (https://www.youtube.com/watch?v=tQpqsmwlfY0) and the Head of Clinical Pharmacology at Moderna saying SciML-based Pumas "has emerged as our 'go-to' tool for most of our analyses in recent months" in 2020 (see https://pumas.ai/). We try to keep a showcase (https://sciml.ai/showcase/) but at this point it's hard to stay on top of the growth. I think anyone would be excited to see an open source side project reach that level of use. Since we tend to focus on core numerical issues (stiffness) and performance, we target the more "hardcore" people in scientific disciplines who really need these aspects and those communities are the ones seeing the most adoption (pharmacology, systems biology, combustion modeling, etc.). Indeed the undergrad classes using a non-stiff ODE solver on small ODEs or training a neural ODE on MNIST don't really have those issues so they aren't our major growth areas. That's okay and that's probably the last group that would move.

In terms of the developer team, throughout the SciML organization repositories we have had around 30 people who have had over 100 commits, which is similar in number to NumPy and SciPy. Julia naturally has a much lower barrier to entry in terms of committing to such packages (since the packages are all in Julia rather than C/Fortran), so the percentage of users who become developers is much higher which is probably why you see a lot more developer activity in contrast to "pure" users. With things like the Python community you have a group of people who write blog posts and teach the tools in courses without ever hacking on the packages or its ecosystem. In Julia, that background is sufficient knowledge to also be developing the package, so everyone writing about Julia seems to also be associated with developing Julia packages somehow. I tend to think that's a positive, but it does make the community look insular as everyone you see writing about Julia is also a developer of packages.

Lastly, since we have been focusing on people with big systems and numerically hard problems, we have had the benefit of being able to overlook some "simple user" issues so far. We are starting to do a big push to clean up things like compile times (https://github.com/SciML/DifferentialEquations.jl/issues/786), improve the documentation, throw better errors, support older versions longer, etc. One way to think about SciML is that it's somewhat the Linux to the monolith Python packages's Windows. We give modular tools in a bunch of different packages that work together, get high performance, and become "more than the sum of the parts", but sometimes people are fine with the simple app made for one purpose. With DEQs, there's a Python package specifically for DEQs (https://github.com/locuslab/deq). Does it have all of the Newton-Krylov choices for the different classes of Jacobians and all of that? No, but it gets something simple and puts an easily Google-able face to it. So while all it takes in Julia with SciML is to stick a nonlinear solver in the right spot in the right way and know how the adjoint codegen will bring it all together, the majority want Visual Studio instead of Awk+Sed or Vim. We understand that, and so the DiffEqFlux.jl package is essentially just a repository of tutorials and prebuilt architectures that people tend to want (https://diffeqflux.sciml.ai/dev/) but we need to continue improving that "simplified experience". The age of Linux is more about making desktop managers that act sufficiently like Windows and less about trying to get everyone building ArchLinux from source. Right now we are currently too much like ArchLinux and need to build more of the Ubuntu-like pieces. We thus have similarly loyal hardcore followers but need to focus a bit on making that installation process easier and the error messages shorter to attract a larger crowd.

> . For example, when we apply convolution filters on images the network consists of repetitive blocks of convolutional layers, and one linear output layer at the very end. It's essentially f(f(f(...f(x))...)) where f is the neural network, and we call this "deep" because of the layers of composition. But what if we make this composition go to infinity?

This really does not jive with my understanding. Each layer of, e.g. VGG 16 [1] does not implement the same function. Each layer has its own weights. There are certain architectures that tie weights across layers, but not all of them.

[1] https://neurohive.io/en/popular-networks/vgg16/

Have you been in interview loops or worked with bread and butter data scientists performing common tasks? I am curious what your view of what most data scientists do day in and day out?

What about tasks being common makes a light-to-moderate understanding of machine learning sufficient?

Processes initiated by data scientists during the execution of their role will tend to fail silently. What is meant here is, throwing an inappropriate model at otherwise good data produces unreliable (catastrophic in certain situations) results, but produces results nonetheless. Without the proper discernment of the reliability of the results, we have an unequivocal failure to execute the role. This is the oft-unmentioned companion to, but decidedly more insidious than, the "garbage in, garbage out" (i.e., right model, wrong data) aphorism.

It is up to the person performing this operation to deduce whether or not the conclusions are trustworthy. I don't see how someone can be confident of this without either relying on a pre-defined workflow verified by someone else qualified to assess the consequences, or to have those qualifications themselves.

What follows is a contrived example, but illustrative of the problem:

Consider e.g. user privacy: it is by now well-known that e.g. embedding vectors (or even merely the relationships between them) can leak a lot of information about the person or object it represents. It is not enough to understand how the forward pass of such a model commences, but also what is stored in those representations, which, having gone through a master's with quite a few people who now call themselves data scientists, I am not confident is commonly understood.

I think it would have helped if the people writing that paper had not confused the issue by introducing a new name for something that is a well known thing in optimal control and had been invented even before neural networks, namely adjoint sensitivity analysis. There even appear multilayer networks of switching components in Pontryagin's book on the subject.