Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Candle: Torch Replacement in Rust

github.com

Post reply on HN
Print view View original

171–180 of 192 posts

Re: Candle: Torch Replacement in Rust

#171

Post by sanderjd »

brrrrrm wrote: post #134
sanderjd wrote: post #127

Earlier quoted context omitted.

Sure, but any language can provide that kind of pre-built block in a library. (Indeed, I suspect the library we're discussing has this operation, though I didn't check.) But if you want to implement a new operation that performs well, not every language can do that. In python, people tend to drop down to C or C++ or increasingly Rust to do that. But it seems (to me) like it would be nice to be able to use a single la…

At least with the more common modern ML (LLMs etc) operations, no one drops down to Rust. It's really only good for IO/CPU work, but that's rarely a spot for new operations, just for data loading and high-latency network stuff. People typically use Triton or CUDA (usually the C++ flavor) to implement operations. It'd be cool if Rust had a CUDA dialect.

The point is that someone dropped down to implement those operations, at some point.

It seems like you can target CUDA with Rust, because this project seems to be doing that?

Re: Candle: Torch Replacement in Rust

#172

Post by riku_iki »

John23832 wrote: post #170
jstx1 wrote: post #29

It's cool that this is coming from huggingface and it isn't just someone's hobby project. Rust is in a weird place where people are trying to push it in a bunch of places where it doesn't belong naturally (web dev, data science) but then when you complain that it's difficult to use you get told that you're using the wrong language for your use case... I really want to like Rust - I've read the Rust book, done the Rus…

I never understood why people say that rust isn’t appropriate for web dev. I think Axum is slick. Typically the people I hear saying that are super Spring indoctrinated.

to me, rust has extra complexity related to memory management and raw performance, which are irrelevant to typical web dev.

I am not an expert in rust, but looking at it as potential choice for my next project.

Re: Candle: Torch Replacement in Rust

#173

Post by infamouscow »

mkehrt wrote: post #157
orangetuba wrote: post #142

Earlier quoted context omitted.

So if you take a Python program and you add the mental effort of having to deal with static types, lifetimes, the borrow checker and performance considerations, and this added mental effort actually allows you to write the program faster? That is incredible. It's like being able to carve- and install a door from a slab of wood faster than installing a prefabricated door.

This line of reasoning is utterly alien to me. If your types add mental overhead, you’re doing something wrong. You should be able to rely on the typechecker to check invariants for you, decreasing mental overhead. You have to think about types anyway, especially in an untyped language, where you can’t rely on the typechecker.

The leap from thinking about your program with inheritance to thinking about your program with traits is large and cannot be overstated.

It's one of the reasons why Python developers don't like working in Ruby, for example.

Re: Candle: Torch Replacement in Rust

#174

Post by brrrrrm »

sanderjd wrote: post #171
brrrrrm wrote: post #134

Earlier quoted context omitted.

At least with the more common modern ML (LLMs etc) operations, no one drops down to Rust. It's really only good for IO/CPU work, but that's rarely a spot for new operations, just for data loading and high-latency network stuff. People typically use Triton or CUDA (usually the C++ flavor) to implement operations. It'd be cool if Rust had a CUDA dialect.

The point is that someone dropped down to implement those operations, at some point. It seems like you can target CUDA with Rust, because this project seems to be doing that?

https://github.com/huggingface/candle/blob/main/candle-kerne...

That’s the C++ variant of cuda, not Rust. They’re just binding it as one might in Python. At which point, what’s the value add?

Re: Candle: Torch Replacement in Rust

#175

Post by jayshua »

liuliu wrote: post #140
GaggiX wrote: post #103

Earlier quoted context omitted.

That's also my workflow but the "Check the data shape, do a eval() run with the data." is the annoying process I was describing, the iterative "waiting for the model to run", "see that it doesn't work", "print random tensor shape trying the debug the code" etc... There is nothing annoying about having an oracle that can check it instantly unless you think that suffering should be part of the experience.

I am not arguing against having an oracle. Just saying in the static type-system (like the ones in Rust / C++ / Swift) would be too frequent to invoke to inhibit development. I think having a linter pass or similar is great. Much like TypeScript actually, where the type hint is the addon rather than intrinsic to the development process, making various things easier at REPL time.

Is this just an issue with the way the editor is presenting the LSP feedback to you?

Re: Candle: Torch Replacement in Rust

#176

Post by zer8k »

jstx1 wrote: post #29

It's cool that this is coming from huggingface and it isn't just someone's hobby project. Rust is in a weird place where people are trying to push it in a bunch of places where it doesn't belong naturally (web dev, data science) but then when you complain that it's difficult to use you get told that you're using the wrong language for your use case... I really want to like Rust - I've read the Rust book, done the Rus…

I was with you until you mentioned Python. The use cases for the two languages are entirely orthogonal. Rust is a fantastic C++ replacement (nothing will replace C).

My problem with Rust, now on month 3 of trying to love it, is exactly what you mentioned. It's a slog. Comparing it to other languages you have to know a lot more about systems. Initially, this isn't a problem, but as it turns out it becomes a problem with Rc/Arc/Box/etc. These are really just leaky abstractions in practice with strange behavior in many cases. Sure, you can write Rust without ever needing that. But if you ever need something more specialized than BTreeMap you're going to be chest deep in a kafkaesque nightmare of lifetimes, weird garbage collection, crazy types (OptionAs long as you can mostly use the standard library and mostly avoid ever needing to do something where resources are shared Rust isn't that bad. Once you start sharing resources your life becomes extremely painful extremely fast. In some cases this is good, requiring you to think, but in many cases it's bad due to the leaky abstractions over memory when you need to do something more. In this case, despite it's major flaws, C++ wins in my opinion. I have a problem with the abstractions over pointers in Rust because they require me to STILL know how things underlying those abstractions works.

Rust is going through its 15 minutes of fame. Much like Ruby, Elixir, etc you will have fans trying to shoehorn every possible case into it. It's a systems language. It is very good at what it is. It definitely does not belong in Data Science though it could be used to replace the underlying systems-level code found in Tensorflow/Torch/Pandas/etc a little safer. But at what cost?

I'm being hard on Rust because I think in 5 years time it'll begin to replace C++ for most new systems level code. I hope they continue to revise the API.

Re: Candle: Torch Replacement in Rust

#177

Post by oivey »

coldtea wrote: post #162
oivey wrote: post #139

Earlier quoted context omitted.

I wish my bugs were mostly preventable via static typing and memory safety checks. For most ML and data work algorithm design and logic errors are the primary source of bugs. Type errors are usually the trivial things you fix in minutes.

The topic wasn't bugs though - and nobody mentioned static typing. The grandparent said "Python is a pain in the ass in production environments". And the parent said: "any programming language can be a pain in production environments". My point was how Rust is better for backend services for fewer problems in production, as it wont have memory bugs (which have long plagued backend services and have resulted in many e…

You’re right, I misread static builds as typing. Python’s dependency management system sure does suck, although I think the problem could be solved without static builds. Overall they are a nice feature, though.

I think the memory management element is a little dubious, though. Torch’s memory management, for example, is fairly safe, and it internally has its own allocator. A Rust version (I think) that implemented the same would have to rely on unsafe blocks, and would likely have many of the same potential pitfalls as Torch. Time spent satisfying Rust’s borrow checker is likely lost productivity compared to just relying on Python’s reference counting and garbage collection. That productivity could be better spent on making the product better with new features and fixes of algorithmic mistakes.

Re: Candle: Torch Replacement in Rust

#178

Post by sanderjd »

brrrrrm wrote: post #174
sanderjd wrote: post #171

Earlier quoted context omitted.

The point is that someone dropped down to implement those operations, at some point. It seems like you can target CUDA with Rust, because this project seems to be doing that?

https://github.com/huggingface/candle/blob/main/candle-kerne... That’s the C++ variant of cuda, not Rust. They’re just binding it as one might in Python. At which point, what’s the value add?

Aha! Thanks for digging that up!

Re: Candle: Torch Replacement in Rust

#179

Post by devjab »

brigadier132 wrote: post #114
devjab wrote: post #67

Earlier quoted context omitted.

Have you written a lot of things in Python? I remember coming "back" to Typescript and C# from Python and how much of a slog it felt. Last year I did quite a lot of work in Rust and I personally found it to be one of the worst languages I've ever worked with in terms of actually writing it. I even like Rust, a lot. I really love things like how you actively need to make things mutable. I wish that was a standard feat…

> Have you written a lot of things in Python? Not the person you responded to but yes. I have written a lot of production software in Python. I remember developing APIs with Python and the day we switched to using Typescript all these bugs we used to have just disappeared. Going from Typescript to Rust was not as big a change but well designed statically typed languages are a productivity booster not killer. We can a…

I don't think we necessarily disagree. I really enjoy using Typescript these days, but I do remember getting into it and being annoyed I had to write {}'s again. These days I'm annoyed when I can't export functions without putting them into a class though, so maybe people shouldn't listen too much to me.

Re: Candle: Torch Replacement in Rust

#180

Post by devjab »

spoiler wrote: post #133
devjab wrote: post #67

Earlier quoted context omitted.

Have you written a lot of things in Python? I remember coming "back" to Typescript and C# from Python and how much of a slog it felt. Last year I did quite a lot of work in Rust and I personally found it to be one of the worst languages I've ever worked with in terms of actually writing it. I even like Rust, a lot. I really love things like how you actively need to make things mutable. I wish that was a standard feat…

> Have you written a lot of things in Python? I've been exposed to Python through work, but I was never a fan. Backwhen I was faced with learning Ruby or Python, I picked Ruby; just felt more ergonomic. I even prefer more recent versions of TypeScript to Python too. I never really felt productive in Python, since the language is very finnicky (I would say more so than JS). Rust might be verbose in parts of the langua…

> Rust might be verbose in parts of the language (but I don't find it as verbose as C or C++), but you get a lot of goodies the the price of a bit of verbosity. So, I don't mind it

I think it's worse than C. That being said, I'm probably also biased in a way where I'm more forgiving to C++ than to Rust because C++ is old and Rust is new. What I found when I was working with Rust was that I wished you'd write it like you write Go. As for Go I quite quickly wished it worked like Rust. So I guess you can't win with me?

These days we mainly use Typescript because it's useful to mainly use a single language in small teams since it allows the one frontend guy to go on vacations without a laptop. We still do some C#, some C and some Rust + the Python which runs our ML/AI/BI parts since that's what those people work with in my area of the world.

Post reply on HN