Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Candle: Torch Replacement in Rust

github.com

Post reply on HN
Print view View original

131–140 of 192 posts

Re: Candle: Torch Replacement in Rust

#131

Post by brundolf »

bombela wrote: post #123
brundolf wrote: post #105

Earlier quoted context omitted.

It'll never not have a trade-off! But there are some things that can make it easier in many cases: - Copy structs- ownership issues vanish when things can be bitwise-copied - .clone()- don't be afraid to clone outside of hot loops! If allocations are a problem, look at using Rc instead of Box where applicable - Pure functions/methods- you don't need mutable references if you aren't mutating things - Macros- reduce bo…

And the implementation behind those abstractions tend to be of really good quality. Thanks to the borrow checker and the scrutiny around the unsafe keyword. In other words I am more willing to trust abstractions over tricky problems in Rust than I would in other languages. Just for a second, try to imagine a generic memoize in C++.

Yep, exactly. The implementation - and more importantly, the contract - are much more dependable. For better and worse Rust is a language that leans all-in on abstraction, going to great lengths to make those as safe and scalable as possible, and if you aren't using them you're going to have a harder time than you need to

Re: Candle: Torch Replacement in Rust

#133

Post by spoiler »

devjab wrote: post #67
spoiler wrote: post #54

Earlier quoted context omitted.

> writing Rust is a slog where you have to jump through hoops and spend a lot more time thinking about programming language abstractions than thinking about the problem you're solving. Have you tried "sticking" with it for a while? Try forcing yoursel to use it[1]. I find that's the best way to learn it (and try reading more code from open source project in Rust you like). I have an opposite experience writing Rust:…

Have you written a lot of things in Python? I remember coming "back" to Typescript and C# from Python and how much of a slog it felt. Last year I did quite a lot of work in Rust and I personally found it to be one of the worst languages I've ever worked with in terms of actually writing it. I even like Rust, a lot. I really love things like how you actively need to make things mutable. I wish that was a standard feat…

> Have you written a lot of things in Python?

I've been exposed to Python through work, but I was never a fan. Backwhen I was faced with learning Ruby or Python, I picked Ruby; just felt more ergonomic. I even prefer more recent versions of TypeScript to Python too. I never really felt productive in Python, since the language is very finnicky (I would say more so than JS).

Rust might be verbose in parts of the language (but I don't find it as verbose as C or C++), but you get a lot of goodies the the price of a bit of verbosity. So, I don't mind it

Re: Candle: Torch Replacement in Rust

#134

Post by brrrrrm »

sanderjd wrote: post #127
brrrrrm wrote: post #44

Earlier quoted context omitted.

It's worth noting that these pre-built blocks are both convenient and incredibly flexible. Before I started getting my hands dirty, it was hard to really see this. For example, here's a masked gather operation along the innermost dimension of an arbitrary rank tensor A: A[..., idx[mask]] I'd hate to have to express that in serial code and then pray a compiler could lift the correct vectorized version out of it.

Sure, but any language can provide that kind of pre-built block in a library. (Indeed, I suspect the library we're discussing has this operation, though I didn't check.) But if you want to implement a new operation that performs well, not every language can do that. In python, people tend to drop down to C or C++ or increasingly Rust to do that. But it seems (to me) like it would be nice to be able to use a single la…

At least with the more common modern ML (LLMs etc) operations, no one drops down to Rust. It's really only good for IO/CPU work, but that's rarely a spot for new operations, just for data loading and high-latency network stuff.

People typically use Triton or CUDA (usually the C++ flavor) to implement operations. It'd be cool if Rust had a CUDA dialect.

Re: Candle: Torch Replacement in Rust

#135

Post by version_five »

I'll say it though I expect violent downmoderation: Rust is way too political for me to ever consider using it for anything real. I've been starting to worry the same about python too (and really all of software) but Rust and Mozilla have that vibe of wanting to go way beyond software development and into identity politics, and it's dangerous to have dependencies with those priorities. Especially in AI because there's an overlap with the "safety" nonsense. New frameworks are always interesting but I wouldn't use rust seriously for anything until the ecosystem broadens enough to incorporate all viewpoints and get away from politics.

Re: Candle: Torch Replacement in Rust

#136

Post by IshKebab »

spoiler wrote: post #54
jstx1 wrote: post #29

It's cool that this is coming from huggingface and it isn't just someone's hobby project. Rust is in a weird place where people are trying to push it in a bunch of places where it doesn't belong naturally (web dev, data science) but then when you complain that it's difficult to use you get told that you're using the wrong language for your use case... I really want to like Rust - I've read the Rust book, done the Rus…

> writing Rust is a slog where you have to jump through hoops and spend a lot more time thinking about programming language abstractions than thinking about the problem you're solving. Have you tried "sticking" with it for a while? Try forcing yoursel to use it[1]. I find that's the best way to learn it (and try reading more code from open source project in Rust you like). I have an opposite experience writing Rust:…

I agree that Rust makes it easier to program... But I'm not sure "you just have to force yourself to use it" is the best advert for Rust!

It's likely that they just haven't got to the point in their programming where they understand the need for the things Rust solves. Like if you only ever write 100 line Python scripts you probably haven't run into the performance issues or difficulties that come with working on large dynamically typed codebases.

Re: Candle: Torch Replacement in Rust

#137

Post by serial_dev »

version_five wrote: post #135

I'll say it though I expect violent downmoderation: Rust is way too political for me to ever consider using it for anything real. I've been starting to worry the same about python too (and really all of software) but Rust and Mozilla have that vibe of wanting to go way beyond software development and into identity politics, and it's dangerous to have dependencies with those priorities. Especially in AI because there'…

Despite all the drama that happens every second month, I'm still "Rust curious". Working with Rust is fun, and I hope it will be my main language in my next job.

You are not the first that noticed the overly political nature of the Rust "community", not a coincidence that we see a blue haired individual in this skit https://youtu.be/TGfQu0bQTKc

I didn't downvote you, but you might be downvoted not because you are wrong, but because it derails the conversation. Turning every Rust thread into a "The Foundation is too woke" can be annoying.

Re: Candle: Torch Replacement in Rust

#138

Post by srush »

unshavedyak wrote: post #70
srush wrote: post #25

Earlier quoted context omitted.

Nowhere near as neat as candle or ggml, but just released a 4-bit rust llama2 implementation with simd. Runs pretty fast. https://github.com/srush/llama2.rs/

That is _really_ cool. Would be interested in some requirements being posted. Eg RAM required for the 70B model if CPU, VRAM required if GPU, etc. edit : I know it's memory mapping, but that still loads data in RAM to execute so if you had 512MiB of RAM it would likely be slow as hell.

Yeah, it's CPU only, and it is using about 38g for 70B and 7g 7B. Guessing that is mostly from the large caches that it keeps around for efficiency. If you wanted to pay some computational cost, you could likely get that down by quantizing activations. Llama2 has some of these tricks built in automatically, for instance grouped query attention.

Re: Candle: Torch Replacement in Rust

#139

Post by oivey »

coldtea wrote: post #95
devjab wrote: post #65

Earlier quoted context omitted.

> Python is a pain in the ass in production environments. I really dislike blanket statements like that one. Because any programming language can be a pain in production environments. One of the disadvantages of Python is also one of it's advantages. Because it's much easier to end up with a terrible production environment when you use "chaos" languages like Python or JavaScript, the tooling to avoid the pain points…

> Because any programming language can be a pain in production environments. Except one with memory safety and static builds. Like Rust can offer.

I wish my bugs were mostly preventable via static typing and memory safety checks. For most ML and data work algorithm design and logic errors are the primary source of bugs. Type errors are usually the trivial things you fix in minutes.

Re: Candle: Torch Replacement in Rust

#140

Post by liuliu »

GaggiX wrote: post #103
liuliu wrote: post #81

Earlier quoted context omitted.

Because that's not my workflow looks like? * Fire up Jupyter notebook, create some PyTorch nn.Module, validate the output is expected. * Built out the model, validate the output is expected. * Check the data shape, do a eval() run with the data. * Move out of Jupyter notebook (or not), to actually run the trainer. Each of the "build out model" phase, shape inference at compile time would be kind of annoying especiall…

That's also my workflow but the "Check the data shape, do a eval() run with the data." is the annoying process I was describing, the iterative "waiting for the model to run", "see that it doesn't work", "print random tensor shape trying the debug the code" etc... There is nothing annoying about having an oracle that can check it instantly unless you think that suffering should be part of the experience.

I am not arguing against having an oracle. Just saying in the static type-system (like the ones in Rust / C++ / Swift) would be too frequent to invoke to inhibit development. I think having a linter pass or similar is great. Much like TypeScript actually, where the type hint is the addon rather than intrinsic to the development process, making various things easier at REPL time.
Post reply on HN