Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Candle: Torch Replacement in Rust

github.com

Post reply on HN
Print view View original

61–70 of 192 posts

Re: Candle: Torch Replacement in Rust

#61

Post by synergy20 »

synergy20 wrote: post #46
jstx1 wrote: post #29

It's cool that this is coming from huggingface and it isn't just someone's hobby project. Rust is in a weird place where people are trying to push it in a bunch of places where it doesn't belong naturally (web dev, data science) but then when you complain that it's difficult to use you get told that you're using the wrong language for your use case... I really want to like Rust - I've read the Rust book, done the Rus…

A similar situation here for me. Rust might be a modern Ada for a particular use case(system programming), I was puzzled why stackoverflow ranked it as 'the most loved' language for years. For each language related post at HN, you will always see rust lover's posts there saying "I wish it's in rust", or "there is a rust version", or "rust is even better". The reality does not match up with the enthusiasm for me thoug…

Talking about my personal experience, getting downvoted heavily which was expected and not a surprise, in fact the insanity is another big reason I will not use rust, no other language has such patterns as far as I can tell.

Re: Candle: Torch Replacement in Rust

#62

Post by zackoverflow »

jstx1 wrote: post #29

It's cool that this is coming from huggingface and it isn't just someone's hobby project. Rust is in a weird place where people are trying to push it in a bunch of places where it doesn't belong naturally (web dev, data science) but then when you complain that it's difficult to use you get told that you're using the wrong language for your use case... I really want to like Rust - I've read the Rust book, done the Rus…

I don't do anything related to data science, but I feel like doing it in Rust would be nice.

You get operator overloading, so you can have ergonomic matrix operations that are typed also. Processing data on the CPU is fast, and crates like https://github.com/EmbarkStudios/rust-gpu make it very ergonomic to leverage the GPU.

I like this library for creating typed coordinate spaces for graphics programming (https://github.com/servo/euclid), I imagine something similar could be done to create refined types for matrices so you don't do matrix multiplication matrices of invalid sizes

Re: Candle: Torch Replacement in Rust

#63

Post by liuliu »

GaggiX wrote: post #53

Hopefully the library will support in the future the shapes of the tensors in the type system, that would be great to prevent many errors.

After a few years, I am coming to an opposite opinion on this. Yeah, shape checking is nice and all and you should be able to run them before-hand (might be a linter pass). But inside type-system while most people have LSP running in the background and bark at you on any type mismatch? It is not going to be a pleasant development experience.

Re: Candle: Torch Replacement in Rust

#64

Post by tikhonj »

dkarl wrote: post #56
jstx1 wrote: post #29

It's cool that this is coming from huggingface and it isn't just someone's hobby project. Rust is in a weird place where people are trying to push it in a bunch of places where it doesn't belong naturally (web dev, data science) but then when you complain that it's difficult to use you get told that you're using the wrong language for your use case... I really want to like Rust - I've read the Rust book, done the Rus…

Rust needs a prototyping/scripting/application language on top of it, with 80% of the speed and 99% of the safety, with automatic memory management (reference counting or GC) and at least one idiomatic way of doing concurrency that disallows sharing mutable state. A language that is tailor-made for building applications quickly on top of the Rust ecosystem, with a build tool that gracefully handles mixed projects if…

I've worked on a Haskell + Rust project that followed roughly the dynamic you're describing and was pretty happy with it. I wrote some simple tooling that let us share algebraic types between the two to manage our interfaces and it worked pretty well. In our case we needed to serialize/deserialize data between our components so the interfaces were at a bit of a distance, but, in principle, it would have been totally doable to extend the same kind of tooling to provide a legitimately easy-to-use FFI layer between the two languages.

Haskell is a bit of a tricky language organizationally for mostly social/political reasons, but if I end up starting my own company in the future, Haskell + Rust is definitely the tech stack I'd lean towards.

Re: Candle: Torch Replacement in Rust

#65

Post by devjab »

wokwokwok wrote: post #41
jstx1 wrote: post #29

It's cool that this is coming from huggingface and it isn't just someone's hobby project. Rust is in a weird place where people are trying to push it in a bunch of places where it doesn't belong naturally (web dev, data science) but then when you complain that it's difficult to use you get told that you're using the wrong language for your use case... I really want to like Rust - I've read the Rust book, done the Rus…

Yeah, but the goal of this isn’t “use this instead of Python to hack up a model” …it’s: > And simply removing Python from production workloads. Python can really add overhead in more complex workflows and the GIL is a notorious source of headaches. You know, those hugging face guys kindaaaaa do know what they’re talking about. Like, really. Python is a pain in the ass in production environments.

> Python is a pain in the ass in production environments.

I really dislike blanket statements like that one. Because any programming language can be a pain in production environments. One of the disadvantages of Python is also one of it's advantages. Because it's much easier to end up with a terrible production environment when you use "chaos" languages like Python or JavaScript, the tooling to avoid the pain points is so mature. With those in place Python becomes great in production.

Yes it'll require your organization to take some serious CI culture decisions and also enforce them. The thing is, however, while you can get a long way without doing that with some programming languages, you're frankly always going to want them by the time you reach enterprise size. So it's just so much easier if you get it done right away, and you will, if you're serious about using Python.

I'd argue that if it's a pain to work in your production environments then the issue is not the technology but the process.

Re: Candle: Torch Replacement in Rust

#66

Post by jahewson »

unshavedyak wrote: post #42
jstx1 wrote: post #29

It's cool that this is coming from huggingface and it isn't just someone's hobby project. Rust is in a weird place where people are trying to push it in a bunch of places where it doesn't belong naturally (web dev, data science) but then when you complain that it's difficult to use you get told that you're using the wrong language for your use case... I really want to like Rust - I've read the Rust book, done the Rus…

It may just be for different people, not sure. I'm a Rust fanatic, i use it for everything, including web dev. We even have a shop full of similar Rust folks. Personally it's my most productive language, and this is after years in Python, NodeJS, and ~5 years Go. Which isn't to attempt to invalidate your experience. But i do think: > writing Rust is a slog where you have to jump through hoops and spend a lot more tim…

Rust for web dev instead of Go? I want to believe. What’s the killer app?

Re: Candle: Torch Replacement in Rust

#67

Post by devjab »

spoiler wrote: post #54
jstx1 wrote: post #29

It's cool that this is coming from huggingface and it isn't just someone's hobby project. Rust is in a weird place where people are trying to push it in a bunch of places where it doesn't belong naturally (web dev, data science) but then when you complain that it's difficult to use you get told that you're using the wrong language for your use case... I really want to like Rust - I've read the Rust book, done the Rus…

> writing Rust is a slog where you have to jump through hoops and spend a lot more time thinking about programming language abstractions than thinking about the problem you're solving. Have you tried "sticking" with it for a while? Try forcing yoursel to use it[1]. I find that's the best way to learn it (and try reading more code from open source project in Rust you like). I have an opposite experience writing Rust:…

Have you written a lot of things in Python? I remember coming "back" to Typescript and C# from Python and how much of a slog it felt. Last year I did quite a lot of work in Rust and I personally found it to be one of the worst languages I've ever worked with in terms of actually writing it.

I even like Rust, a lot. I really love things like how you actively need to make things mutable. I wish that was a standard feature in any language I work with. But it's just so verbose.

Re: Candle: Torch Replacement in Rust

#68

Post by tracker1 »

unshavedyak wrote: post #42
jstx1 wrote: post #29

It's cool that this is coming from huggingface and it isn't just someone's hobby project. Rust is in a weird place where people are trying to push it in a bunch of places where it doesn't belong naturally (web dev, data science) but then when you complain that it's difficult to use you get told that you're using the wrong language for your use case... I really want to like Rust - I've read the Rust book, done the Rus…

It may just be for different people, not sure. I'm a Rust fanatic, i use it for everything, including web dev. We even have a shop full of similar Rust folks. Personally it's my most productive language, and this is after years in Python, NodeJS, and ~5 years Go. Which isn't to attempt to invalidate your experience. But i do think: > writing Rust is a slog where you have to jump through hoops and spend a lot more tim…

I'd like to make the jump to FT Rust myself, just don't have enough hands on... and as I get older, motivation to work on more stuff on the side, as well as time, becomes more limited.

Re: Candle: Torch Replacement in Rust

#69

Post by locuscoeruleus »

spenczar5 wrote: post #3
tomrod wrote: post #2

Oh man. I feel like this is a really big deal. I've been wondering what might move data scientists to Rust. Outside working through rustlings and trying new projects in Rust, I wonder if having this available will be it.

I agree. Two years ago, I tried writing data-intensive components for Python as rust extension modules. It was painful, tedious, and confusing. I came back to it for a second try a month ago, and was blown away. PyO3 and maturin ( especially maturin) are stunningly straightforward. I actually find maturin easier to use than “normal” Python packaging tools, to the extent such a thing even exists. I really hope we will…

How can I effectively approach extending Python with Rust? I encountered some challenges while attempting to translate Python objects to Rust. My intuition suggests that it's more straightforward to work with arrays or dictionaries of primitive types. Could you share your approach and perhaps point me to some helpful examples or resources for better understanding?

Re: Candle: Torch Replacement in Rust

#70

Post by unshavedyak »

srush wrote: post #25
ianpurton wrote: post #9

I think the first Rust ML framework to gain massive traction will be the one that supports quantisation out of the box. At the moment there's no Rust ML library that does what llama.cpp or ggml can do.

Nowhere near as neat as candle or ggml, but just released a 4-bit rust llama2 implementation with simd. Runs pretty fast. https://github.com/srush/llama2.rs/

That is _really_ cool. Would be interested in some requirements being posted. Eg RAM required for the 70B model if CPU, VRAM required if GPU, etc.

edit: I know it's memory mapping, but that still loads data in RAM to execute so if you had 512MiB of RAM it would likely be slow as hell.

Post reply on HN