Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Lm.rs: Minimal CPU LLM inference in Rust with no dependency

github.com

Post reply on HN
Print view View original

21–30 of 79 posts

Re: Lm.rs: Minimal CPU LLM inference in Rust with no dependency

#21

Post by simonw »

amelius wrote: post #18
simonw wrote: post #15

This is impressive. I just ran the 1.2G llama3.2-1b-it-q80.lmrs on a M2 64GB MacBook and it felt speedy and used 1000% of CPU across 13 threads (according to Activity Monitor). cd /tmp git clone https://github.com/samuel-vitorino/lm.rs cd lm.rs RUSTFLAGS="-C target-cpu=native" cargo build --release --bin chat curl -LO 'https://huggingface.co/samuel-vitorino/Llama-3.2-1B-Instruct-Q8_0-LMRS/resolve/main/tokenizer.bin?d…

Not sure how to formulate this, but what does this mean in the sense of how "smart" it is compared to the latest chatgpt version?

The model I'm running here is Llama 3.2 1B, the smallest on-device model I've tried that has given me good results.

The fact that a 1.2GB download can do as well as this is honestly astonishing to me - but it's going to laughably poor in comparison to something like GPT-4o - which I'm guessing is measured in the 100s of GBs.

You can try out Llama 3.2 1B yourself directly in your browser (it will fetch about 1GB of data) at https://chat.webllm.ai/

Re: Lm.rs: Minimal CPU LLM inference in Rust with no dependency

#22

Post by littlestymaar »

amelius wrote: post #18
simonw wrote: post #15

This is impressive. I just ran the 1.2G llama3.2-1b-it-q80.lmrs on a M2 64GB MacBook and it felt speedy and used 1000% of CPU across 13 threads (according to Activity Monitor). cd /tmp git clone https://github.com/samuel-vitorino/lm.rs cd lm.rs RUSTFLAGS="-C target-cpu=native" cargo build --release --bin chat curl -LO 'https://huggingface.co/samuel-vitorino/Llama-3.2-1B-Instruct-Q8_0-LMRS/resolve/main/tokenizer.bin?d…

Not sure how to formulate this, but what does this mean in the sense of how "smart" it is compared to the latest chatgpt version?

The implementation has no control on “how smart” the model is, and when it comes to llama 1B, it's not very smart by current standard (but it would still have blown everyone's mind just a few years back).

Re: Lm.rs: Minimal CPU LLM inference in Rust with no dependency

#23

Post by simonw »

littlestymaar wrote: post #17
simonw wrote: post #15

This is impressive. I just ran the 1.2G llama3.2-1b-it-q80.lmrs on a M2 64GB MacBook and it felt speedy and used 1000% of CPU across 13 threads (according to Activity Monitor). cd /tmp git clone https://github.com/samuel-vitorino/lm.rs cd lm.rs RUSTFLAGS="-C target-cpu=native" cargo build --release --bin chat curl -LO 'https://huggingface.co/samuel-vitorino/Llama-3.2-1B-Instruct-Q8_0-LMRS/resolve/main/tokenizer.bin?d…

Could you try with ./target/release/chat --model llama3.2-1b-it-q80.lmrs --show-metrics To know how many token/s you get?

Nice, just tried that with "tell me a long tall tale" as the prompt and got:

    Speed: 26.41 tok/s
Full output: https://gist.github.com/simonw/6f25fca5c664b84fdd4b72b091854...

Re: Lm.rs: Minimal CPU LLM inference in Rust with no dependency

#24

Post by KeplerBoy »

littlestymaar wrote: post #22
amelius wrote: post #18

Earlier quoted context omitted.

Not sure how to formulate this, but what does this mean in the sense of how "smart" it is compared to the latest chatgpt version?

The implementation has no control on “how smart” the model is, and when it comes to llama 1B, it's not very smart by current standard (but it would still have blown everyone's mind just a few years back).

The implementation absolutely can influence the outputs.

If you have a sloppy implementations which somehow accumulates a lot of error in it's floating point math, you will get worse results.

It's rarely talked about, but it's a real thing. Floating point addition and multiplication is non-associative and the order of operations affects the correctness and performance. Developers might (unknowningly) trade performance for correctness. And it matters a lot more in the low precision modes we operate today. Just try different methods of summing a vector containing 9,999 fp16 ones in fp16. Hint: it will never be 9,999.0 and you won't get close to the best approximation if you do it in a naive loop.

Re: Lm.rs: Minimal CPU LLM inference in Rust with no dependency

#25

Post by littlestymaar »

KeplerBoy wrote: post #24
littlestymaar wrote: post #22

Earlier quoted context omitted.

The implementation has no control on “how smart” the model is, and when it comes to llama 1B, it's not very smart by current standard (but it would still have blown everyone's mind just a few years back).

The implementation absolutely can influence the outputs. If you have a sloppy implementations which somehow accumulates a lot of error in it's floating point math, you will get worse results. It's rarely talked about, but it's a real thing. Floating point addition and multiplication is non-associative and the order of operations affects the correctness and performance. Developers might (unknowningly) trade performanc…

TIL, thanks.

Re: Lm.rs: Minimal CPU LLM inference in Rust with no dependency

#26

Post by J_Shelby_J »

littlestymaar wrote: post #19
saghm wrote: post #13

Earlier quoted context omitted.

The readme seems to indicate that it expects pytorch alongside several other Python dependencies in a requirements.txt file (which is the only place I can find any form of the word "dependency" on the page). I'm very confused by the characterization in the title here given that it doesn't seem to be claimed at all by the project itself (which simple has the subtitle "Minimal LLM inference in Rust"). From the git hist…

> The readme seems to indicate that it expects pytorch alongside several other Python dependencies in a requirements.txt file That's only if you want to convert the model yourself, you don't need that if you use the converted weights on the author's huggingface page (in “prepared-models” table of the README). > From the git history, it looks like the username of the person who posted this here is someone who has cont…

What do you think about implementing your gui for other rust LLM projects? I’m looking for a front end for my project: https://github.com/ShelbyJenkins/llm_client

Re: Lm.rs: Minimal CPU LLM inference in Rust with no dependency

#27

Post by J_Shelby_J »

littlestymaar wrote: post #16
wyldfire wrote: post #4

The title is less clear than it could be IMO. When I saw "no dependency" I thought maybe it could be no_std (llama.c is relatively lightweight in this regard). But it's definitely not `no_std` and in fact seems like it has several dependencies. Perhaps all of them are rust dependencies?

Titles are hard. What I wanted to express is that it doesn't have any pytorch or Cuda or onnx or whatever deep learning dependency and that all the logic is self contained. To be totally transparent it has 5 Rust dependencies by default, two of them should be feature gated for the chat (chrono and clap), and then there are 3 utility crates that are used to get a little bit more performance out of the hardware (`rayon…

Yeah, hard to not be overly verbose. “No massive dependencies with long build times and deep abstractions!” Is not as catchy.

Re: Lm.rs: Minimal CPU LLM inference in Rust with no dependency

#28

Post by ctz »

wyldfire wrote: post #4

The title is less clear than it could be IMO. When I saw "no dependency" I thought maybe it could be no_std (llama.c is relatively lightweight in this regard). But it's definitely not `no_std` and in fact seems like it has several dependencies. Perhaps all of them are rust dependencies?

The original may have made sense, eg "no hardware dependency", or "no GPU dependency". Unfortunately HN deletes words from titles with no rhyme or reason, and no transparency.

Re: Lm.rs: Minimal CPU LLM inference in Rust with no dependency

#29

Post by sroussey »

KeplerBoy wrote: post #24
littlestymaar wrote: post #22

Earlier quoted context omitted.

The implementation has no control on “how smart” the model is, and when it comes to llama 1B, it's not very smart by current standard (but it would still have blown everyone's mind just a few years back).

The implementation absolutely can influence the outputs. If you have a sloppy implementations which somehow accumulates a lot of error in it's floating point math, you will get worse results. It's rarely talked about, but it's a real thing. Floating point addition and multiplication is non-associative and the order of operations affects the correctness and performance. Developers might (unknowningly) trade performanc…

How well does bf16 work in comparison?

Re: Lm.rs: Minimal CPU LLM inference in Rust with no dependency

#30

Post by 0x457 »

J_Shelby_J wrote: post #27
littlestymaar wrote: post #16

Earlier quoted context omitted.

Titles are hard. What I wanted to express is that it doesn't have any pytorch or Cuda or onnx or whatever deep learning dependency and that all the logic is self contained. To be totally transparent it has 5 Rust dependencies by default, two of them should be feature gated for the chat (chrono and clap), and then there are 3 utility crates that are used to get a little bit more performance out of the hardware (`rayon…

Yeah, hard to not be overly verbose. “No massive dependencies with long build times and deep abstractions!” Is not as catchy.

No dependencies in this case (and pretty much any rust project) means: to build you need rustc+cargo and to use you just need resulting binary.

As in you don't need to have C compiler, python, dynamic libraries. "pure rust" would be a better way to describe it.

Post reply on HN