how does this compare to https://github.com/EricLBuehler/mistral.rs ?
Lm.rs: Minimal CPU LLM inference in Rust with no dependency
51–60 of 79 posts
Re: Lm.rs: Minimal CPU LLM inference in Rust with no dependency
#52Earlier quoted context omitted.
The model I'm running here is Llama 3.2 1B, the smallest on-device model I've tried that has given me good results. The fact that a 1.2GB download can do as well as this is honestly astonishing to me - but it's going to laughably poor in comparison to something like GPT-4o - which I'm guessing is measured in the 100s of GBs. You can try out Llama 3.2 1B yourself directly in your browser (it will fetch about 1GB of da…
anyone else think 4o is kinda garbage compared to the older gpt4? as well as o1-preview and probably o1-mini. gpt4 tends to be more accurate than 4o for me.
Yes, I use 4o for customer support in multiple languages and sometimes I have to tell it to reply using the customer language, while gpt4 could easily infer it.
Re: Lm.rs: Minimal CPU LLM inference in Rust with no dependency
#53This is impressive. I just ran the 1.2G llama3.2-1b-it-q80.lmrs on a M2 64GB MacBook and it felt speedy and used 1000% of CPU across 13 threads (according to Activity Monitor). cd /tmp git clone https://github.com/samuel-vitorino/lm.rs cd lm.rs RUSTFLAGS="-C target-cpu=native" cargo build --release --bin chat curl -LO 'https://huggingface.co/samuel-vitorino/Llama-3.2-1B-Instruct-Q8_0-LMRS/resolve/main/tokenizer.bin?d…
Not sure how to formulate this, but what does this mean in the sense of how "smart" it is compared to the latest chatgpt version?
[deleted]
Re: Lm.rs: Minimal CPU LLM inference in Rust with no dependency
#54What's the value of this compared to llama.cpp?
Easier to integrate with other Rust projects maybe?
Re: Lm.rs: Minimal CPU LLM inference in Rust with no dependency
#55Great! Did something similar some time ago [0] but the performance was underwhelming compared to C/C++ code running on CPU (which points to my lack of understanding of how to make Rust fast). Would be nice to have some benchmarks of the different Rust implementations. Implementing LLM inference should/could really become the new "hello world!" for serious programmers out there :) [0] https://github.com/gip/yllama.rs
i also had a similar 'hello world' experience some time ago with [0] :). i manually used some SIMD instructions, and it seems the performance could align with llama.cpp. it appears that the key to performance is:
1. using SIMD on quantized matrix multiplication 2. using a busy loop instead of condition variables when splitting work among threads.
(however, i haven't had more free time to continue working on inferencing quantized models on GPU (with Vulkan), and it hasn't been updated for a long time since then.)
Re: Lm.rs: Minimal CPU LLM inference in Rust with no dependency
#56This is beautifully written, thanks for sharing. I could see myself using some of the source code in the classroom to explain how transformers "really" work; code is more concrete/detailed than all those pictures of attention heads etc. Two points of minor criticism/suggestions for improvement: - libraries should not print to stdout, as that output may detroy application output (imagine I want to use the library in a…
> best to write to a string buffer It's best to call a user callback. That way logs can be, for example, displayed in a GUI.
A good logging framework has all the hooks you need
Re: Lm.rs: Minimal CPU LLM inference in Rust with no dependency
#57This is cool (and congrats on writing your first Rust lib!), but Metal/Cuda support is a must for serious local usage.
Re: Lm.rs: Minimal CPU LLM inference in Rust with no dependency
#58Interesting, I appreciate the rust community‘s enthu to rewrite most the stuff.
Re: Lm.rs: Minimal CPU LLM inference in Rust with no dependency
#59This is cool (and congrats on writing your first Rust lib!), but Metal/Cuda support is a must for serious local usage.
Using Cuda is a non starter because it would go against the purpose of this project, but I (not the main author but contributor) am experimenting with wgpu to get some kind of GPU acceleration.
I'm not sure it goes anywhere though, because the main author want to keep the complexity under control.
Re: Lm.rs: Minimal CPU LLM inference in Rust with no dependency
#60how does this compare to https://github.com/EricLBuehler/mistral.rs ?
Much simpler codebase because it has much less features. It doesn't aim to be a llama.cpp competitor AFAIK.