Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Lm.rs: Minimal CPU LLM inference in Rust with no dependency

github.com

Post reply on HN
Print view View original

11–20 of 79 posts

Re: Lm.rs: Minimal CPU LLM inference in Rust with no dependency

#11

Post by J_Shelby_J »

lucgagan wrote: post #5

Correct me if I am wrong, but these implementations are all CPU bound?, i.e. if I have a good GPU, I should look for alternatives.

Yes. Depending on gpu 10-20x difference.

For rust you have the llama.cpp wrappers like llm_client (mine), and the candle based projects mistral.rs, and Kalosm.

Although, my project does try and provide an implementation of mistral.rs, I haven’t fully migrated from llama.cpp. A full rust implementation would be nice for quick install times (among other reasons). Right now my crate has to clone and build. It’s automated for mac, pc, and Linux but it adds about a minute of build time.

Re: Lm.rs: Minimal CPU LLM inference in Rust with no dependency

#12

Post by gip »

Great! Did something similar some time ago [0] but the performance was underwhelming compared to C/C++ code running on CPU (which points to my lack of understanding of how to make Rust fast). Would be nice to have some benchmarks of the different Rust implementations.

Implementing LLM inference should/could really become the new "hello world!" for serious programmers out there :)

[0] https://github.com/gip/yllama.rs

Re: Lm.rs: Minimal CPU LLM inference in Rust with no dependency

#13

Post by saghm »

wyldfire wrote: post #4

The title is less clear than it could be IMO. When I saw "no dependency" I thought maybe it could be no_std (llama.c is relatively lightweight in this regard). But it's definitely not `no_std` and in fact seems like it has several dependencies. Perhaps all of them are rust dependencies?

The readme seems to indicate that it expects pytorch alongside several other Python dependencies in a requirements.txt file (which is the only place I can find any form of the word "dependency" on the page). I'm very confused by the characterization in the title here given that it doesn't seem to be claimed at all by the project itself (which simple has the subtitle "Minimal LLM inference in Rust").

From the git history, it looks like the username of the person who posted this here is someone who has contributed to the project but isn't the primary author. If they could elaborate on what exactly they mean by saying this has "zero dependencies", that might be helpful.

Re: Lm.rs: Minimal CPU LLM inference in Rust with no dependency

#14

Post by vitaminka »

wyldfire wrote: post #4

The title is less clear than it could be IMO. When I saw "no dependency" I thought maybe it could be no_std (llama.c is relatively lightweight in this regard). But it's definitely not `no_std` and in fact seems like it has several dependencies. Perhaps all of them are rust dependencies?

is rust cargo basically like npm at this point? like how on earth is sixteen dependencies means no dependencies lol

Re: Lm.rs: Minimal CPU LLM inference in Rust with no dependency

#15

Post by simonw »

This is impressive. I just ran the 1.2G llama3.2-1b-it-q80.lmrs on a M2 64GB MacBook and it felt speedy and used 1000% of CPU across 13 threads (according to Activity Monitor).

    cd /tmp
    git clone https://github.com/samuel-vitorino/lm.rs
    cd lm.rs
    RUSTFLAGS="-C target-cpu=native" cargo build --release --bin chat
    curl -LO 'https://huggingface.co/samuel-vitorino/Llama-3.2-1B-Instruct-Q8_0-LMRS/resolve/main/tokenizer.bin?download=true'
    curl -LO 'https://huggingface.co/samuel-vitorino/Llama-3.2-1B-Instruct-Q8_0-LMRS/resolve/main/llama3.2-1b-it-q80.lmrs?download=true'
    ./target/release/chat --model llama3.2-1b-it-q80.lmrs

Re: Lm.rs: Minimal CPU LLM inference in Rust with no dependency

#16

Post by littlestymaar »

wyldfire wrote: post #4

The title is less clear than it could be IMO. When I saw "no dependency" I thought maybe it could be no_std (llama.c is relatively lightweight in this regard). But it's definitely not `no_std` and in fact seems like it has several dependencies. Perhaps all of them are rust dependencies?

Titles are hard.

What I wanted to express is that it doesn't have any pytorch or Cuda or onnx or whatever deep learning dependency and that all the logic is self contained.

To be totally transparent it has 5 Rust dependencies by default, two of them should be feature gated for the chat (chrono and clap), and then there are 3 utility crates that are used to get a little bit more performance out of the hardware (`rayon` for easier parallelization, `wide` for helping with SIMD, and `memmap2` for memory mapping of the model file).

Re: Lm.rs: Minimal CPU LLM inference in Rust with no dependency

#17

Post by littlestymaar »

simonw wrote: post #15

This is impressive. I just ran the 1.2G llama3.2-1b-it-q80.lmrs on a M2 64GB MacBook and it felt speedy and used 1000% of CPU across 13 threads (according to Activity Monitor). cd /tmp git clone https://github.com/samuel-vitorino/lm.rs cd lm.rs RUSTFLAGS="-C target-cpu=native" cargo build --release --bin chat curl -LO 'https://huggingface.co/samuel-vitorino/Llama-3.2-1B-Instruct-Q8_0-LMRS/resolve/main/tokenizer.bin?d…

Could you try with

    ./target/release/chat --model llama3.2-1b-it-q80.lmrs --show-metrics
To know how many token/s you get?

Re: Lm.rs: Minimal CPU LLM inference in Rust with no dependency

#18

Post by amelius »

simonw wrote: post #15

This is impressive. I just ran the 1.2G llama3.2-1b-it-q80.lmrs on a M2 64GB MacBook and it felt speedy and used 1000% of CPU across 13 threads (according to Activity Monitor). cd /tmp git clone https://github.com/samuel-vitorino/lm.rs cd lm.rs RUSTFLAGS="-C target-cpu=native" cargo build --release --bin chat curl -LO 'https://huggingface.co/samuel-vitorino/Llama-3.2-1B-Instruct-Q8_0-LMRS/resolve/main/tokenizer.bin?d…

Not sure how to formulate this, but what does this mean in the sense of how "smart" it is compared to the latest chatgpt version?

Re: Lm.rs: Minimal CPU LLM inference in Rust with no dependency

#19

Post by littlestymaar »

saghm wrote: post #13
wyldfire wrote: post #4

The title is less clear than it could be IMO. When I saw "no dependency" I thought maybe it could be no_std (llama.c is relatively lightweight in this regard). But it's definitely not `no_std` and in fact seems like it has several dependencies. Perhaps all of them are rust dependencies?

The readme seems to indicate that it expects pytorch alongside several other Python dependencies in a requirements.txt file (which is the only place I can find any form of the word "dependency" on the page). I'm very confused by the characterization in the title here given that it doesn't seem to be claimed at all by the project itself (which simple has the subtitle "Minimal LLM inference in Rust"). From the git hist…

> The readme seems to indicate that it expects pytorch alongside several other Python dependencies in a requirements.txt file

That's only if you want to convert the model yourself, you don't need that if you use the converted weights on the author's huggingface page (in “prepared-models” table of the README).

> From the git history, it looks like the username of the person who posted this here is someone who has contributed to the project but isn't the primary author.

Yup that's correct, so far I've only authored the dioxus GUI app.

> If they could elaborate on what exactly they mean by saying this has "zero dependencies", that might be helpful.

See my other response: https://news.ycombinator.com/item?id=41812665

Re: Lm.rs: Minimal CPU LLM inference in Rust with no dependency

#20

Post by littlestymaar »

vitaminka wrote: post #14
wyldfire wrote: post #4

The title is less clear than it could be IMO. When I saw "no dependency" I thought maybe it could be no_std (llama.c is relatively lightweight in this regard). But it's definitely not `no_std` and in fact seems like it has several dependencies. Perhaps all of them are rust dependencies?

is rust cargo basically like npm at this point? like how on earth is sixteen dependencies means no dependencies lol

> like how on earth is sixteen dependencies means no dependencies lol

You're counting optional dependencies used in the binaries which isn't fair (obviously the GUI app or the backend of the webui are going to have dependencies!). But yes 3 dependencies isn't literally no dependency.

Post reply on HN