We have optimized neural ODEs, neural SDEs, neural DDEs, and neural DAEs quite a bit in Julia, and have some experimental results with neural jump SDEs and neural PDEs. But we can really do more with the neural PDEs, especially neural PDEs with constraints (i.e. partial differential-algebraic equations) and neural stochastic PDEs (SPDEs). It's all working with DiffEqFlux.jl (you can see a blog post of experiments here:
http://www.stochasticlifestyle.com/neural-jump-sdes-jump-dif...), and it works with GPUs, but we really need to make a few of the overloads like mapslices have better GPU kernels to really capture the neural SPDE applications well.
I'm also spending quite a bit of time looking into the ability for these neural differential equations to understand phase space. What exactly do they learn, and why? Can you predict beforehand how well you can learn the equations directly from data? Of course, this is intertwined with the previous part, because to understand how well you can auto-learn neural SPDEs you need the ability to quickly solve neural SPDEs, along with all of the implicit solvers and GPU support.