Rust isn't going to be used until it has better CUDA support.
Python vs. Rust for Neural Networks
141–149 of 149 posts
Re: Python vs. Rust for Neural Networks
#142Earlier quoted context omitted.
This is just not true. The python runtime is not the bottleneck. DL frameworks are DSLs written on top of piles of highly optimized C++ code that is executed as independently from the python runtime as possible. Optimizing the python or swapping it out for some other language is not going to buy you anything except a ton of work. We can argue about using rust to implement the lower level ops instead of c++. That migh…
The python runtime is not the bottleneck. This smells like an overgeneralization. Often things that aren’t a bottleneck in the context of the problems you’ve faced might at least be an unacceptable cost in the context of the 16.6 ms budget someone else is working within.
In what circumstance would one measure end to end time budget in training? What would that metric tell you? You don't care about latency, you care about throughput, which can be scaled nearly completely independently of the "wrapper language" for lack of a better term, in this case that's python.
It seems some commenters on this thread have not really thought through the lifecycle of a learned model and the tradeoffs existing frameworks exploit to make things fast _and_ easy to use. In training we care about throughput. Thats great because we can use a high level DSL to construct some graph that trains in a highly concurrent execution mode or on dedicated hardware. Using the high level DSL is what allows us to abstract away these details and still get good training throughput. Tradeoffs still bleed out of the abstraction (think batch size, network size, architecture etc have effect on how efficient certain hardware will be) but that is inevitable when you're moving from CPU to GPU to ASIC.
When you are done training and you want to use the model in a low latency environment you use a c/c++ binary to serve it. Latency matters there, so exploit the fact that you're no longer defining a model (no need for a fancy DSL) and just serve it from a very simple but highly optimized API.
Re: Python vs. Rust for Neural Networks
#143Earlier quoted context omitted.
A dynamic language can be super frustrating to develop with because you have to keep tensor dimensions memorized in your head or in comments
How is à statically typed language going to help with tensor dimensions?
Named/typed axes checking - ie. I shouldn't be able to contract along two dimensions of different types (like batch dimension contracted with time dimension)
Re: Python vs. Rust for Neural Networks
#144Nobody writing NN in Python, they are just describing it. For NN or DL in general, the correctness doesn't really lie too much on the code quality level, like ownership Rust people love to talk about. It is more about Numeric stability under/overflow and such. Choice of programming language offers limited help here. I don't think Rust has a killer app for ML/DL community to offer as of now, the focus is vastly differ…
Rust people will talk about ownership, because when you work with the language you learn to appreciate what it implies. Rust code for ml can be fairly effortlessly compiled to any target - including wasm. Not so for python running over c-compiled libs. Rust has the potential to NN over a bunch of phones and laptops all collaborating on the same thing together. Huge potential, esp around dapps that need a little ml in their lives.
Re: Python vs. Rust for Neural Networks
#145Earlier quoted context omitted.
This is why I think Swift is a much better choice to replace or at least compliment Python than Rust. It has a modern, powerful type system and all the quality of life advantages which come with it, but it manages this with a lot more usability than Rust. A well written swift framework almost becomes a DSL for the problem domain, which is a great property for a data science tool to have.
Swift is not fun on non-Mac platforms. It feels a lot like Google's Dart in terms of how it was positioned and advocated. Modern Rust isn't difficult to use. This is becoming a really tired meme from detractors. The compiler is incredibly helpful, non-lexical lifetimes are a thing, and unless you're doing a lot of sharing and parallelism, you can avoid many borrow checker problems until you learn RAII.
The "Rust is difficult" criticism isn't getting tired, but it must be getting tiresome for the Rust community because they don't have a good answer to it.
The top level comment from an early adopter in the last Rust release thread on HN was complaining about the tedious complexity.
Re: Python vs. Rust for Neural Networks
#146Earlier quoted context omitted.
Yes. Let’s say you want certain features of a voice sample. You need to do that feature engineering every time before you send it to the model. Doesn’t it make sense to do it in C++ or Rust? This is currently already done. So if you already are starting to do parts of the feature engineering in Rust why not continue? Yeah it’s not reasonable right now because Python has the best ecosystem. But that will not always be…
I can’t exactly tell what you mean but I think you’re confusing two levels of abstraction here. C++ (or rust) and python already work in harmony to make training efficient. 1. In tensorflow and similar frameworks the Python runtime is used to compose highly optimized operations to create a trainable graph. 2. C++ is used to implement those highly optimized ops. If you have some novel feature engineering and you need…
I know. That’s how we do it too. You don’t see any benefits in instead of Python wrappers + C++ just do Rust? Especially in handling large data like voice iff there was a good ecosystem and toolbox in place?
Re: Python vs. Rust for Neural Networks
#147Earlier quoted context omitted.
CUDA is an important part of the story. I think the industry is moving to 'MLIR' solution (Yes, there is a Google project called exactly that, but I am referring to the general idea here), where the network is defined and trained in one place, then the weights are exported, delegated to optimized runtime to be executed. If such trend furthers down, then there will be very little reason to replace Python as the glue l…
Python should and will be replaced, but not at all for any of the reasons mentioned in this thread. A good ML language is going to need smart and static typing. I am so tired of having to run a whole network just to figure out that there's a dimension mismatch because I forgot to take a transpose somewhere - there is essentially no reason that tensor shapes can't just be inferred and these errors caught pre-runtime.
Normal static typing won't help you there. You would need some sort of dependent typing. For example, look at Idris.
Re: Python vs. Rust for Neural Networks
#148Earlier quoted context omitted.
I can’t exactly tell what you mean but I think you’re confusing two levels of abstraction here. C++ (or rust) and python already work in harmony to make training efficient. 1. In tensorflow and similar frameworks the Python runtime is used to compose highly optimized operations to create a trainable graph. 2. C++ is used to implement those highly optimized ops. If you have some novel feature engineering and you need…
I know. That’s how we do it too. You don’t see any benefits in instead of Python wrappers + C++ just do Rust? Especially in handling large data like voice iff there was a good ecosystem and toolbox in place?
Maybe but then we’re no longer making an argument about performance, which is what I was responding to in your initial claim about “everything counts” and numpy shuffle being slow. That’s a straw man argument that has zero bearing on actual engineering decisions.
EDIT: clarification in first sentence
Re: Python vs. Rust for Neural Networks
#149Earlier quoted context omitted.
A bit snarky isn’t it? I explained why it isn’t irrational to use Rust — nowhere did I say you have to use Rust or that Rust is the only solution to that problem (which it obviously isn’t). Assuming the person opposite doesn’t know things is an extra unpleasant move on your part and I am not sure why you feel attacked by any of what I wrote. For the record, I also use C and I know that expert level C can beat naive R…
Sounds like you’re over reading into what was essentially a two-point comment... there was no snarkiness in that comment at all - just a statement that rust is not really offering much benefit for this circumstance. Yeesh... projection
Sorry for that, it was entirely unnecessary — as you rightly noticed I read something into the comment that wasn’t there.