Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Python vs. Rust for Neural Networks

ngoldbaum.github.io

Post reply on HN
Print view View original

111–120 of 149 posts

Re: Python vs. Rust for Neural Networks

#111

Post by ddragon »

MiroF wrote: post #95
xxxpupugo wrote: post #28

Earlier quoted context omitted.

CUDA is an important part of the story. I think the industry is moving to 'MLIR' solution (Yes, there is a Google project called exactly that, but I am referring to the general idea here), where the network is defined and trained in one place, then the weights are exported, delegated to optimized runtime to be executed. If such trend furthers down, then there will be very little reason to replace Python as the glue l…

Python should and will be replaced, but not at all for any of the reasons mentioned in this thread. A good ML language is going to need smart and static typing. I am so tired of having to run a whole network just to figure out that there's a dimension mismatch because I forgot to take a transpose somewhere - there is essentially no reason that tensor shapes can't just be inferred and these errors caught pre-runtime.

Do you have an example of a tensor library that keep track of shapes and detect mismatches at compile time? I had the impression that even in static languages having tensors with the exact shape as a parameter would stress the compiler, forcing it to compile many versions of every function for every possible size combination, and the output of a function could very well have a non deterministic or multiple possible shapes (for example branching on runtime information). So they compromise and make only the dimensionality as a parameter, which would not catch your example either until the runtime bound checks.

Re: Python vs. Rust for Neural Networks

#112

Post by danielscrubs »

kuzehanka wrote: post #105
danielscrubs wrote: post #88

Earlier quoted context omitted.

Every small thing counts when you have big data which is exactly why you need performance everywhere, if Rust can help with that I don’t mind switching my team to that. The problem are usually when you do novel feature engineering not the actual model training. But I was a C++ dev before checking the assembly for performance optimization so I guess I have more wiggle room to see when things are not up to snuff. If I…

Novel feature engineering? Like this? https://towardsdatascience.com/python-performance-and-gpus-1...

Looks good. I’ve tried Numba and that was extremely limited.

Current project we can’t use GPUs for production so we can only use it for development. Not my call, but operations. They have a Kubernetes cluster and a take it or leave it attitude.

We did end up using C++ for somethings and Python for most. I’d feel comfortable with C++ or Rust alone if there was a great ecosystem for DS though.

Re: Python vs. Rust for Neural Networks

#113

Post by HelloNurse »

In the end, hypothetical future good Rust libraries for linear algebra will just be turned into Python extensions and embraced into the mainstream.

They'll also, equally well, serve the needs of Rust programs which need serious number crunching (presumably a small niche); there is no "versus" in the comparison.

Re: Python vs. Rust for Neural Networks

#114

Post by TheRealKing »

ianamartin wrote: post #17
psv1 wrote: post #10

Neural network libraries (Tensorflow, Pytorch) have a C++ backend and a Python interface. Which is great - you get a performant compiled language as the backend and a flexible user-friendly language as the interface. Rust vs Python is a weird question because in reality no one writes their own neural network with numpy, and no one expects Rust to act like an interpreted language suitable for data science workflows. I…

Yeah, this is the real point. Python is an interface to C, C++, and FORTRAN for a lot of stuff. There are even crossover libs for running R. This is like comparing apples and steaks.

"Fortran", since 1990 (FORTRAN refers to F77 :-)

Re: Python vs. Rust for Neural Networks

#115

Post by dnautics »

corndoge wrote: post #72
kuzehanka wrote: post #65

Earlier quoted context omitted.

I must be missing something. Modern data science workloads involve fanning out data and code across dozens to hundreds of nodes. The bottlenecks, in order, are: inter-node comms, gpu/compute, on-disk shuffling, serialisation, pipeline starvation, and finally the runtime. Why worry about optimising the very top of the perf pyramid which will make the least difference? Why worry if you spent 1ms pushing data to numpy w…

Good lord, hopefully latency isn't 2.5 seconds!

quote is 'data spend 2500 ms on the wire'. That's not latency. For a nice 10GbE connection, that's optimistically 3 GB or so worth of data. Do you have 3GB of training data? Then it will spend 2500 ms on the wire to distribute to all of your nodes as part of startup.

Re: Python vs. Rust for Neural Networks

#116

Post by MiroF »

ddragon wrote: post #111
MiroF wrote: post #95

Earlier quoted context omitted.

Python should and will be replaced, but not at all for any of the reasons mentioned in this thread. A good ML language is going to need smart and static typing. I am so tired of having to run a whole network just to figure out that there's a dimension mismatch because I forgot to take a transpose somewhere - there is essentially no reason that tensor shapes can't just be inferred and these errors caught pre-runtime.

Do you have an example of a tensor library that keep track of shapes and detect mismatches at compile time? I had the impression that even in static languages having tensors with the exact shape as a parameter would stress the compiler, forcing it to compile many versions of every function for every possible size combination, and the output of a function could very well have a non deterministic or multiple possible s…

If you explain a little more of what you mean, I might be able to respond more effectively.

> I had the impression that even in static languages having tensors with the exact shape as a parameter would stress the compiler, forcing it to compile many versions of every function for every possible size combination, and the output of a function could very well have a non deterministic or multiple possible shapes (for example branching on runtime information).

I was a bit lazy in my original comment - you're right. What I really think should be implemented (and is already starting to in Pytorch and a library named NamedTensor, albeit non-statically) is essentially having "typed axes."

For instance, if I had a sequence of locations in time, I could describe the tensor as:

(3 : DistanceAxis, 32 : TimeAxis, 32 : BatchAxis).

Sure, the number of dimensions could vary and you're right that, if so, the approach implied by my first comment would have a combinatorial explosion. But if I'm contracting a TimeAxis with a BatchAxis accidentally, that can be pretty easily caught before I even have to run the code. But in normal pytorch, such a contraction would succeed - and it would succeed silently.

Re: Python vs. Rust for Neural Networks

#117

Post by danielscrubs »

MiroF wrote: post #107
danielscrubs wrote: post #103

Earlier quoted context omitted.

or maybe, just maybe, you know nothing of optimizations, so you just go: impossible!

the optimization you are describing is premature - you don't need rust to productionize your models and in most cases you don't even need to be coding in a low-level language at all.

I don’t think I’ll change your mind.

You clearly have more experience than I do that can solve everything in Python instead of C++ and/or CUDA.

You win.

Re: Python vs. Rust for Neural Networks

#118

Post by ahurmazda »

I could see how Rust could be a good option for writing the prediction server. In my use case, DL models are trained offline. It does not matter if it’s faster by 30 mins in training (maybe when I R&D-ing).

On the other hand, you typically have to reach for something like c++ for a low latency, high thruput environment. It can be a bear to write a server. Anything more ergonomic would be welcome

Re: Python vs. Rust for Neural Networks

#119

Post by MiroF »

danielscrubs wrote: post #117
MiroF wrote: post #107

Earlier quoted context omitted.

the optimization you are describing is premature - you don't need rust to productionize your models and in most cases you don't even need to be coding in a low-level language at all.

I don’t think I’ll change your mind. You clearly have more experience than I do that can solve everything in Python instead of C++ and/or CUDA. You win.

I mean, it'd be easier to change my mind if you had a single reason behind anything you've claimed.

I would love to understand why Rust would be more effective for productionizing ML models than the existing infrastructure written in Python/C++/CUDA.

Re: Python vs. Rust for Neural Networks

#120

Post by danielscrubs »

kahnjw wrote: post #108
danielscrubs wrote: post #88

Earlier quoted context omitted.

Every small thing counts when you have big data which is exactly why you need performance everywhere, if Rust can help with that I don’t mind switching my team to that. The problem are usually when you do novel feature engineering not the actual model training. But I was a C++ dev before checking the assembly for performance optimization so I guess I have more wiggle room to see when things are not up to snuff. If I…

This is just not true. The python runtime is not the bottleneck. DL frameworks are DSLs written on top of piles of highly optimized C++ code that is executed as independently from the python runtime as possible. Optimizing the python or swapping it out for some other language is not going to buy you anything except a ton of work. We can argue about using rust to implement the lower level ops instead of c++. That migh…

Yes. Let’s say you want certain features of a voice sample. You need to do that feature engineering every time before you send it to the model. Doesn’t it make sense to do it in C++ or Rust? This is currently already done. So if you already are starting to do parts of the feature engineering in Rust why not continue?

Yeah it’s not reasonable right now because Python has the best ecosystem. But that will not always be the case!

Post reply on HN