Earlier quoted context omitted.
>there's nothing stopping you from composing things in Python. In Julia you can compose a custom distribution, with a bayesian model with an ODE with a neural network with unit number types with custom Julia written CUDA kernels and multithreading. Edit: That are not designed specifically to work with each other Can python even hope to do a fraction of that, still be fast and differentiate through everything?
I'm fairly certain everything you said is possible except for custom CUDA kernels in pure Python. You'd have to write the kernel in C++ and use it in your TensorFlow/PyTorch code. [0][1] [0]: https://www.tensorflow.org/guide/create_op [1]: https://pytorch.org/tutorials/advanced/cpp_extension.html
Even then, you're not going to have fast custom types to be used on the GPU without dropping into C++