Interesting. The repo was owned by Laurent Mazare[0] a few days ago, I'm wondering if HuggingFace just hired them to work on candle full time?
[0]: Also maintainer of https://github.com/LaurentMazare/tch-rs
Candle: Torch Replacement in Rust
71–80 of 192 posts
Re: Candle: Torch Replacement in Rust
#72Looks great! Rust is the wrong language for developing ML models, but it has a clear potential for being the best one for deploying them.
Re: Candle: Torch Replacement in Rust
#73Earlier quoted context omitted.
Yeah, but the goal of this isn’t “use this instead of Python to hack up a model” …it’s: > And simply removing Python from production workloads. Python can really add overhead in more complex workflows and the GIL is a notorious source of headaches. You know, those hugging face guys kindaaaaa do know what they’re talking about. Like, really. Python is a pain in the ass in production environments.
Pytorch has multiple production paths for deployment models without the python interpreter already, like torchscript and libtorch, or more annoying paths like onnx export and onnx runtime. So there is no need to have rust to solve this problem.
Those are great technologies for making the model inference independent of Python. However, there's always a good deal of pre and post processing, or other business logic, that needs to go around model inference. This sort of thing needs to be done in a general programming language. It's in these situations where Python is often used (because the supporting model code is often written by the same person, and that code is probably going to be Python because your model training and eval code is most likely going to be Python too).
This is where having a non-Python PL, such as Rust, can make a huge difference for both ease of production deployment/maintenance as well as having truely efficient production inference.
Re: Candle: Torch Replacement in Rust
#74Earlier quoted context omitted.
It may just be for different people, not sure. I'm a Rust fanatic, i use it for everything, including web dev. We even have a shop full of similar Rust folks. Personally it's my most productive language, and this is after years in Python, NodeJS, and ~5 years Go. Which isn't to attempt to invalidate your experience. But i do think: > writing Rust is a slog where you have to jump through hoops and spend a lot more tim…
Rust for web dev instead of Go? I want to believe. What’s the killer app?
> Rust for web dev instead of Go? I want to believe. What’s the killer app?
No killer app here. We just prefer Rust itself. Go is very capable and a great choice.
For me personally i found Go to spread logic out far too much. Some features of Rust, Iterators for example, massively simplified how i think about data flow. That was my #1 issue with Go. This has side root-causes in Go of course, like no generics to allow for Iterators, etc - but these days Go has generics so it's likely improving. Features around Enums are also huge for me.
I left Go ages ago and it's improved a lot sense then. I still prefer Rust, though.
As i said no judgement on any lang, including Go. This is merely my experience and preferences.
Re: Candle: Torch Replacement in Rust
#75Hopefully the library will support in the future the shapes of the tensors in the type system, that would be great to prevent many errors.
After a few years, I am coming to an opposite opinion on this. Yeah, shape checking is nice and all and you should be able to run them before-hand (might be a linter pass). But inside type-system while most people have LSP running in the background and bark at you on any type mismatch? It is not going to be a pleasant development experience.
I cannot really see how it can be a worse experience than waiting for the model to run, fix the error found, wait a second time, fix another error; when you can just check instantly if the model architecture is sound.
Re: Candle: Torch Replacement in Rust
#76Earlier quoted context omitted.
It may just be for different people, not sure. I'm a Rust fanatic, i use it for everything, including web dev. We even have a shop full of similar Rust folks. Personally it's my most productive language, and this is after years in Python, NodeJS, and ~5 years Go. Which isn't to attempt to invalidate your experience. But i do think: > writing Rust is a slog where you have to jump through hoops and spend a lot more tim…
Rust for web dev instead of Go? I want to believe. What’s the killer app?
I would say for me, the killer app would be just about anything you might reach for Electron with, but want a much smaller install footprint. Tauri in and of itself is pretty cool, kind of a lighter alternative to electron using native platform browser engine.
In general, I would probably reach for Rust if I needed to do anything wither Web-Assembly (wasm) as the path to working is much nicer than other languages imo. Along those lines, Yew is pretty nice as a rust-like framework.
I've also played with Rocket, and found it very nice and effective for api driven web-apps. Haven't done any templating engines in/with Rocket, however.
Re: Candle: Torch Replacement in Rust
#77It's cool that this is coming from huggingface and it isn't just someone's hobby project. Rust is in a weird place where people are trying to push it in a bunch of places where it doesn't belong naturally (web dev, data science) but then when you complain that it's difficult to use you get told that you're using the wrong language for your use case... I really want to like Rust - I've read the Rust book, done the Rus…
> writing Rust is a slog where you have to jump through hoops and spend a lot more time thinking about programming language abstractions than thinking about the problem you're solving.
It's like this until it isn't. Eventually this falls completely away - I promise.
It's a lot like being a 100% backend engineer trying to develop in React for the first time. You just have to get used to it.
Rust is not difficult, it just takes some soaking in.
Re: Candle: Torch Replacement in Rust
#78"Torch replacement"? Title should be "Candle: Minimalist ML framework for Rust" if anything.
Re: Candle: Torch Replacement in Rust
#79It's cool that this is coming from huggingface and it isn't just someone's hobby project. Rust is in a weird place where people are trying to push it in a bunch of places where it doesn't belong naturally (web dev, data science) but then when you complain that it's difficult to use you get told that you're using the wrong language for your use case... I really want to like Rust - I've read the Rust book, done the Rus…
Prototype in Python, but scale in Rust. I do a lot of AI Engineering in Python, but production Data Engineering, and Feature Engineering workloads are done in parallel with Rust. Simple and Fast. You can even make custom-made python modules with a Rust core when Numpy matrices are not enough to crack your graph-traversing problem efficiently ! (Thanks PyO3 and Maturin)
> Prototype in Python, but scale in Rust.
Not even. Some people start building big systems with Java or Go. It's entirely appropriate to start in Rust too once you're more proficient in it.
Re: Candle: Torch Replacement in Rust
#80Exciting tech, looking forward to see it running on wgpu! There is an issue about WebGPU although I don’t notice references to the native version of it.