Lm.rs: Minimal CPU LLM inference in Rust with no dependency
1–10 of 79 posts
Re: Lm.rs: Minimal CPU LLM inference in Rust with no dependency
#2Nice work, it would be great to see some benchmarks comparing it to llm.c.
Re: Lm.rs: Minimal CPU LLM inference in Rust with no dependency
#3This is really cool.
It's already using Dioxus (neat). I wonder if WASM could be put on the roadmap.
If this could run a lightweight LLM like RWKV in the browser, then the browser unlocks a whole class of new capabilities without calling any SaaS APIs.
Re: Lm.rs: Minimal CPU LLM inference in Rust with no dependency
#4The title is less clear than it could be IMO.
When I saw "no dependency" I thought maybe it could be no_std (llama.c is relatively lightweight in this regard). But it's definitely not `no_std` and in fact seems like it has several dependencies. Perhaps all of them are rust dependencies?
Re: Lm.rs: Minimal CPU LLM inference in Rust with no dependency
#5Correct me if I am wrong, but these implementations are all CPU bound?, i.e. if I have a good GPU, I should look for alternatives.
Re: Lm.rs: Minimal CPU LLM inference in Rust with no dependency
#6Correct me if I am wrong, but these implementations are all CPU bound?, i.e. if I have a good GPU, I should look for alternatives.
[deleted]
Re: Lm.rs: Minimal CPU LLM inference in Rust with no dependency
#7Neat.
FYI I have a whole bunch of rust tools[0] for loading models and other LLM tasks. For example auto selecting the largest quant based on memory available, extracting a tokenizer from a gguf, prompting, etc. You could use this to remove some of the python dependencies you have.
Currently to support llama.cpp, but this is pretty neat too. Any plans to support grammars?
Re: Lm.rs: Minimal CPU LLM inference in Rust with no dependency
#8This is really cool. It's already using Dioxus (neat). I wonder if WASM could be put on the roadmap. If this could run a lightweight LLM like RWKV in the browser, then the browser unlocks a whole class of new capabilities without calling any SaaS APIs.
I was poking at this a bit here
https://github.com/maedoc/rwkv.js
using the Rwkv.cpp compiled with emscripten, but I didn’t quite figure out the tokenizers part (yet, only spent about an hour on it)
Nevertheless I am pretty sure the 1.6b rwkv6 would be totally usable offline browser only. It’s not capable enough for general chat but for rag etc it could be quite enough
Re: Lm.rs: Minimal CPU LLM inference in Rust with no dependency
#9Correct me if I am wrong, but these implementations are all CPU bound?, i.e. if I have a good GPU, I should look for alternatives.
It's all implemented on the CPU, yes, there's no GPU acceleration whatsoever (at the moment at least).
> if I have a good GPU, I should look for alternatives.
If you actually want to run it, even just on the CPU, you should look for an alternative (and the alternative is called llama.cpp) this is more of an educational resource about how things work when you remove all the layers of complexity in the ecosystem.
LLM are somewhat magic in how effective they can be, but in terms of code it's really simple.
Re: Lm.rs: Minimal CPU LLM inference in Rust with no dependency
#10Correct me if I am wrong, but these implementations are all CPU bound?, i.e. if I have a good GPU, I should look for alternatives.
You are correct. This project is "on the CPU", so it will not utilize your GPU for computation. If you would like to try out a Rust framework that does support GPUs, Candle https://github.com/huggingface/candle/tree/main may be worth exploring