Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Lm.rs: Minimal CPU LLM inference in Rust with no dependency

github.com

Post reply on HN
Print view View original

31–40 of 79 posts

Re: Lm.rs: Minimal CPU LLM inference in Rust with no dependency

#31

Post by KeplerBoy »

sroussey wrote: post #29
KeplerBoy wrote: post #24

Earlier quoted context omitted.

The implementation absolutely can influence the outputs. If you have a sloppy implementations which somehow accumulates a lot of error in it's floating point math, you will get worse results. It's rarely talked about, but it's a real thing. Floating point addition and multiplication is non-associative and the order of operations affects the correctness and performance. Developers might (unknowningly) trade performanc…

How well does bf16 work in comparison?

Even worse, I'd say since it has fewer bits for the fraction. At least in the example i was mentioning, where you run into precision limits, not into range limits.

I believe bf16 was primarily designed as a storage format, since it just needs 16 zero bits added to be a valid fp32.

Re: Lm.rs: Minimal CPU LLM inference in Rust with no dependency

#32

Post by jll29 »

This is beautifully written, thanks for sharing.

I could see myself using some of the source code in the classroom to explain how transformers "really" work; code is more concrete/detailed than all those pictures of attention heads etc.

Two points of minor criticism/suggestions for improvement:

- libraries should not print to stdout, as that output may detroy application output (imagine I want to use the library in a text editor to offer style checking). So best to write to a string buffer owned by a logging class instance associated with a lm.rs object.

- Is it possible to do all this without "unsafe" without twisting one's arm? I see there are uses of "unsafe" e.g. to force data alignment in the model reader.

Again, thanks and very impressive!

Re: Lm.rs: Minimal CPU LLM inference in Rust with no dependency

#33

Post by littlestymaar »

echelon wrote: post #3

This is really cool. It's already using Dioxus (neat). I wonder if WASM could be put on the roadmap. If this could run a lightweight LLM like RWKV in the browser, then the browser unlocks a whole class of new capabilities without calling any SaaS APIs.

> I wonder if WASM could be put on the roadmap.

The library itself should be able to compile to WASM with very little change: rayon and wide the only mandatory dependencies support wasm out of the box, and to get rid of memmap2 by replacing the `Mmap` type in transformer.rs with `&[u8]`.

That being said, RWKV is a completely different architecture so it should be reimplemented entierly and is not likely to be part of the roadmap ever (not the main author so I can't say for sure, but I really doubt it).

Re: Lm.rs: Minimal CPU LLM inference in Rust with no dependency

#34

Post by tormeh »

vitaminka wrote: post #14
wyldfire wrote: post #4

The title is less clear than it could be IMO. When I saw "no dependency" I thought maybe it could be no_std (llama.c is relatively lightweight in this regard). But it's definitely not `no_std` and in fact seems like it has several dependencies. Perhaps all of them are rust dependencies?

is rust cargo basically like npm at this point? like how on earth is sixteen dependencies means no dependencies lol

Yes, basically. Someone who is a dependency maximalist (never write any code that can be replaced by a dependency) then you can easily end up with a thousand dependencies. I don't like things being that way, but others do.

It's worth noting that Rust's std library is really small, and you therefore need more dependencies in Rust than in some other languages like Python. There are some "blessed" crates though, like the ones maintained by the rust-lang team themselves (https://crates.io/teams/github:rust-lang:libs and https://crates.io/teams/github:rust-lang-nursery:libs). Also, when you add a dependency like Tokio, Axum, or Polars, these are often ecosystems of crates rather than singular crates.

Tl;dr: Good package managers end up encouraging micro-dependencies and dependency bloat because these things are now painless. Cargo is one of these good package managers.

Re: Lm.rs: Minimal CPU LLM inference in Rust with no dependency

#36

Post by littlestymaar »

0x457 wrote: post #30
J_Shelby_J wrote: post #27

Earlier quoted context omitted.

Yeah, hard to not be overly verbose. “No massive dependencies with long build times and deep abstractions!” Is not as catchy.

No dependencies in this case (and pretty much any rust project) means: to build you need rustc+cargo and to use you just need resulting binary. As in you don't need to have C compiler, python, dynamic libraries. "pure rust" would be a better way to describe it.

It's a little bit more than pure Rust: to build the library there's basically only two dependencies (rayon and wide) which bring only 14 transitive dependencies (anyone who's built even simple Rust program knows that this is a very small number).

And there's more, Rayon and wide are only needed for performance and we could trivially put them behind a feature flag and have zero dependency and have the library work in a no-std context actually, but it would be so slow it would have no use at all so I don't really think that makes sense to do except in order to win an argument…

Re: Lm.rs: Minimal CPU LLM inference in Rust with no dependency

#37

Post by littlestymaar »

fuddle wrote: post #2

Nice work, it would be great to see some benchmarks comparing it to llm.c.

I doubt it would compare favorably at the moment, I don't think it's particularly well optimized besides using rayon to get CPU parallelism and wide for a bit of SIMD.

It's good enough to get pretty good performance for little effort, but I don't think it would win a benchmark race either.

Re: Lm.rs: Minimal CPU LLM inference in Rust with no dependency

#38

Post by jiggawatts »

KeplerBoy wrote: post #24
littlestymaar wrote: post #22

Earlier quoted context omitted.

The implementation has no control on “how smart” the model is, and when it comes to llama 1B, it's not very smart by current standard (but it would still have blown everyone's mind just a few years back).

The implementation absolutely can influence the outputs. If you have a sloppy implementations which somehow accumulates a lot of error in it's floating point math, you will get worse results. It's rarely talked about, but it's a real thing. Floating point addition and multiplication is non-associative and the order of operations affects the correctness and performance. Developers might (unknowningly) trade performanc…

I thought all current implementations accumulate into a fp32 instead of accumulating in fp16.

Re: Lm.rs: Minimal CPU LLM inference in Rust with no dependency

#39

Post by jll29 »

tormeh wrote: post #34
vitaminka wrote: post #14

Earlier quoted context omitted.

is rust cargo basically like npm at this point? like how on earth is sixteen dependencies means no dependencies lol

Yes, basically. Someone who is a dependency maximalist (never write any code that can be replaced by a dependency) then you can easily end up with a thousand dependencies. I don't like things being that way, but others do. It's worth noting that Rust's std library is really small, and you therefore need more dependencies in Rust than in some other languages like Python. There are some "blessed" crates though, like th…

How about designing a "proper" standard library for Rust (comparable to Java's or CommonLISP's), to ensure a richer experience, avoiding dependency explosions, and also to ensure things are written in a uniform interface style? Is that something the Rust folks are considering or actively working on?

EDIT: nobody is helped by 46 regex libraries, none of which implements Unicode fully, for example (not an example taken from the Rust community).

Re: Lm.rs: Minimal CPU LLM inference in Rust with no dependency

#40

Post by tormeh »

jll29 wrote: post #39
tormeh wrote: post #34

Earlier quoted context omitted.

Yes, basically. Someone who is a dependency maximalist (never write any code that can be replaced by a dependency) then you can easily end up with a thousand dependencies. I don't like things being that way, but others do. It's worth noting that Rust's std library is really small, and you therefore need more dependencies in Rust than in some other languages like Python. There are some "blessed" crates though, like th…

How about designing a "proper" standard library for Rust (comparable to Java's or CommonLISP's), to ensure a richer experience, avoiding dependency explosions, and also to ensure things are written in a uniform interface style? Is that something the Rust folks are considering or actively working on? EDIT: nobody is helped by 46 regex libraries, none of which implements Unicode fully, for example (not an example taken…

Just use the rust-lang org's regex crate. It's fascinating that you managed to pick one of like 3 high-level use-cases that are covered by official rust-lang crates.
Post reply on HN