Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Lm.rs: Minimal CPU LLM inference in Rust with no dependency

github.com

Post reply on HN
Print view View original

61–70 of 79 posts

Re: Lm.rs: Minimal CPU LLM inference in Rust with no dependency

#61

Post by janwas »

jiggawatts wrote: post #38
KeplerBoy wrote: post #24

Earlier quoted context omitted.

The implementation absolutely can influence the outputs. If you have a sloppy implementations which somehow accumulates a lot of error in it's floating point math, you will get worse results. It's rarely talked about, but it's a real thing. Floating point addition and multiplication is non-associative and the order of operations affects the correctness and performance. Developers might (unknowningly) trade performanc…

I thought all current implementations accumulate into a fp32 instead of accumulating in fp16.

We (gemma.cpp) recently started accumulating softmax terms into f64. There is at least one known case of this causing differing output, but after 200 tokens, hence unlikely to be detected in many benchmarks.

Does anyone have experience with higher-precision matmul and whether it is worthwhile?

Re: Lm.rs: Minimal CPU LLM inference in Rust with no dependency

#63

Post by jodleif »

simonw wrote: post #23
littlestymaar wrote: post #17

Earlier quoted context omitted.

Could you try with ./target/release/chat --model llama3.2-1b-it-q80.lmrs --show-metrics To know how many token/s you get?

Nice, just tried that with "tell me a long tall tale" as the prompt and got: Speed: 26.41 tok/s Full output: https://gist.github.com/simonw/6f25fca5c664b84fdd4b72b091854...

How much with llama.cpp? A 1b model should be a lot faster on a m2

Re: Lm.rs: Minimal CPU LLM inference in Rust with no dependency

#64

Post by amelius »

azinman2 wrote: post #56
LoganDark wrote: post #42

Earlier quoted context omitted.

> best to write to a string buffer It's best to call a user callback. That way logs can be, for example, displayed in a GUI.

A good logging framework has all the hooks you need

Doesn't rust have a standard solution for that?

If I use 10 libraries and they all use a different logging framework then that's ... not convenient.

Re: Lm.rs: Minimal CPU LLM inference in Rust with no dependency

#65

Post by sapiogram »

amelius wrote: post #64
azinman2 wrote: post #56

Earlier quoted context omitted.

A good logging framework has all the hooks you need

Doesn't rust have a standard solution for that? If I use 10 libraries and they all use a different logging framework then that's ... not convenient.

It does, everyone uses the `log` crate. But then it wouldn't be zero-dependencies anymore.

Re: Lm.rs: Minimal CPU LLM inference in Rust with no dependency

#66

Post by MuffinFlavored »

simonw wrote: post #21
amelius wrote: post #18

Earlier quoted context omitted.

Not sure how to formulate this, but what does this mean in the sense of how "smart" it is compared to the latest chatgpt version?

The model I'm running here is Llama 3.2 1B, the smallest on-device model I've tried that has given me good results. The fact that a 1.2GB download can do as well as this is honestly astonishing to me - but it's going to laughably poor in comparison to something like GPT-4o - which I'm guessing is measured in the 100s of GBs. You can try out Llama 3.2 1B yourself directly in your browser (it will fetch about 1GB of da…

> that has given me good results.

Can you help somebody out of the loop frame/judge/measure 'good results'?

Can you give an example of something it can do that's impressive/worthwhile? Can you give an example of where it falls short / gets tripped up?

Is it just a hallucination machine? What good does that do for anybody? Genuinely trying to understand.

Re: Lm.rs: Minimal CPU LLM inference in Rust with no dependency

#67

Post by littlestymaar »

sapiogram wrote: post #65
amelius wrote: post #64

Earlier quoted context omitted.

Doesn't rust have a standard solution for that? If I use 10 libraries and they all use a different logging framework then that's ... not convenient.

It does, everyone uses the `log` crate. But then it wouldn't be zero-dependencies anymore.

In fairness it's already not really “zero dependency” since it uses rayon (for easy multithreading) and wide (for easy SIMD), using log would make total sense I think (not the main author, just a contributor).

Re: Lm.rs: Minimal CPU LLM inference in Rust with no dependency

#68

Post by tormeh »

littlestymaar wrote: post #59
dvt wrote: post #57

This is cool (and congrats on writing your first Rust lib!), but Metal/Cuda support is a must for serious local usage.

Using Cuda is a non starter because it would go against the purpose of this project, but I (not the main author but contributor) am experimenting with wgpu to get some kind of GPU acceleration. I'm not sure it goes anywhere though, because the main author want to keep the complexity under control.

wgpu would be awesome. Too little ML software out there is hardware-agnostic.

Re: Lm.rs: Minimal CPU LLM inference in Rust with no dependency

#69

Post by tormeh »

lucgagan wrote: post #5

Correct me if I am wrong, but these implementations are all CPU bound?, i.e. if I have a good GPU, I should look for alternatives.

Depends. Good models are big, and require a lot of memory. Even the 4090 doesn't have that much memory in an LLM context. So your GPU will be faster, but likely can't fit the big models.

Re: Lm.rs: Minimal CPU LLM inference in Rust with no dependency

#70

Post by gspr »

vitaminka wrote: post #14
wyldfire wrote: post #4

The title is less clear than it could be IMO. When I saw "no dependency" I thought maybe it could be no_std (llama.c is relatively lightweight in this regard). But it's definitely not `no_std` and in fact seems like it has several dependencies. Perhaps all of them are rust dependencies?

is rust cargo basically like npm at this point? like how on earth is sixteen dependencies means no dependencies lol

Indeed. It's the one cultural aspect of Rust I find exhausting. Huge fan of the language and the community in general, but a few widespread attitudes do drive me nuts:

* That adding dependencies is something you should take very lightly

* The everybody uses or should use crates.io for dependencies

* That it's OK to just ask users to use the latest release of something at all times

* That vendoring code is always a good thing when it adds even the slightest convenience

* That one should ship generated code (prominent in e.g. crates that use FFI bindings)

* The idea that as long as software doesn't depend on something non-Rust, it doesn't have dependencies

Luckily the language, the standard library and the community in general are of excellent quality.

Post reply on HN