Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Lm.rs: Minimal CPU LLM inference in Rust with no dependency

github.com

Post reply on HN
Print view View original

41–50 of 79 posts

Re: Lm.rs: Minimal CPU LLM inference in Rust with no dependency

#41

Post by KeplerBoy »

jiggawatts wrote: post #38
KeplerBoy wrote: post #24

Earlier quoted context omitted.

The implementation absolutely can influence the outputs. If you have a sloppy implementations which somehow accumulates a lot of error in it's floating point math, you will get worse results. It's rarely talked about, but it's a real thing. Floating point addition and multiplication is non-associative and the order of operations affects the correctness and performance. Developers might (unknowningly) trade performanc…

I thought all current implementations accumulate into a fp32 instead of accumulating in fp16.

I haven't looked at all implementations, but the hardware (tensor cores as well as cuda cores) allows you to accumulate at fp16 precision.

Re: Lm.rs: Minimal CPU LLM inference in Rust with no dependency

#42

Post by LoganDark »

jll29 wrote: post #32

This is beautifully written, thanks for sharing. I could see myself using some of the source code in the classroom to explain how transformers "really" work; code is more concrete/detailed than all those pictures of attention heads etc. Two points of minor criticism/suggestions for improvement: - libraries should not print to stdout, as that output may detroy application output (imagine I want to use the library in a…

> best to write to a string buffer

It's best to call a user callback. That way logs can be, for example, displayed in a GUI.

Re: Lm.rs: Minimal CPU LLM inference in Rust with no dependency

#43

Post by iknowstuff »

simonw wrote: post #21
amelius wrote: post #18

Earlier quoted context omitted.

Not sure how to formulate this, but what does this mean in the sense of how "smart" it is compared to the latest chatgpt version?

The model I'm running here is Llama 3.2 1B, the smallest on-device model I've tried that has given me good results. The fact that a 1.2GB download can do as well as this is honestly astonishing to me - but it's going to laughably poor in comparison to something like GPT-4o - which I'm guessing is measured in the 100s of GBs. You can try out Llama 3.2 1B yourself directly in your browser (it will fetch about 1GB of da…

anyone else think 4o is kinda garbage compared to the older gpt4? as well as o1-preview and probably o1-mini.

gpt4 tends to be more accurate than 4o for me.

Re: Lm.rs: Minimal CPU LLM inference in Rust with no dependency

#44

Post by pornel »

jll29 wrote: post #39
tormeh wrote: post #34

Earlier quoted context omitted.

Yes, basically. Someone who is a dependency maximalist (never write any code that can be replaced by a dependency) then you can easily end up with a thousand dependencies. I don't like things being that way, but others do. It's worth noting that Rust's std library is really small, and you therefore need more dependencies in Rust than in some other languages like Python. There are some "blessed" crates though, like th…

How about designing a "proper" standard library for Rust (comparable to Java's or CommonLISP's), to ensure a richer experience, avoiding dependency explosions, and also to ensure things are written in a uniform interface style? Is that something the Rust folks are considering or actively working on? EDIT: nobody is helped by 46 regex libraries, none of which implements Unicode fully, for example (not an example taken…

The particular mode of distribution of code as a traditional standard library has downsides:

- it's inevitably going to accumulate mistakes/obsolete/deprecated stuff over time, because there can be only one version of it, and it needs to be backwards compatible.

- it makes porting the language to new platforms harder, since there's more stuff promised to work as standard.

- to reduce risk of having the above problems, stdlib usually sticks to basic lowest-common-denominator APIs, lagging behind the state of the art, creating a dilemma between using standard impl vs better but 3rd party impls (and large programs end up with both)

- with a one-size-fits-all it's easy to add bloat from unnecessary features. Not all programs want to embed megabytes of Unicode metadata for a regex.

The goal of having common trustworthy code can be achieved in many other ways, such as having (de-facto) standard individual dependencies to choose from. Packages that aren't built-in can be versioned independently, and included only when necessary.

Re: Lm.rs: Minimal CPU LLM inference in Rust with no dependency

#48

Post by airstrike »

iknowstuff wrote: post #43
simonw wrote: post #21

Earlier quoted context omitted.

The model I'm running here is Llama 3.2 1B, the smallest on-device model I've tried that has given me good results. The fact that a 1.2GB download can do as well as this is honestly astonishing to me - but it's going to laughably poor in comparison to something like GPT-4o - which I'm guessing is measured in the 100s of GBs. You can try out Llama 3.2 1B yourself directly in your browser (it will fetch about 1GB of da…

anyone else think 4o is kinda garbage compared to the older gpt4? as well as o1-preview and probably o1-mini. gpt4 tends to be more accurate than 4o for me.

I sort of do, especially against OG GPT-4 (before turbo)

4o is a bit too lobotomized for my taste. If you try to engage in conversation, nearly every answer after the first starts with "You're absolutely right". Bro, I don't know if I'm right, that's why I'm asking a question!

It's somehow better in _some_ scenarios but I feel like it's also objectively worse in others so it ends up being a wash. It paradoxically looks bad relative to GPT-4 but also makes GPT-4 feel worse when you go back to it...

o1-preview has been growing on me despite its answers also being very formulaic (relative to the OG GPT-3.5 and GPT-4 models which had more "freedom" in how they answered)

Re: Lm.rs: Minimal CPU LLM inference in Rust with no dependency

#49

Post by kkielhofner »

lucgagan wrote: post #5

Correct me if I am wrong, but these implementations are all CPU bound?, i.e. if I have a good GPU, I should look for alternatives.

CPU, yes, but more importantly memory bandwidth.

An RTX 3090 (as one example) has nearly 1TB/s of memory bandwidth. You'd need at least 12 channels of the fastest proof-of-concept DDR5 on the planet to equal that.

If you have a discrete GPU, use an implementation that utilizes it because it's a completely different story.

Apple Silicon boasts impressive numbers on LLM inference because it has a unified CPU-GPU high-bandwidth (400GB/s IIRC) memory architecture.

Re: Lm.rs: Minimal CPU LLM inference in Rust with no dependency

#50

Post by csomar »

iknowstuff wrote: post #43
simonw wrote: post #21

Earlier quoted context omitted.

The model I'm running here is Llama 3.2 1B, the smallest on-device model I've tried that has given me good results. The fact that a 1.2GB download can do as well as this is honestly astonishing to me - but it's going to laughably poor in comparison to something like GPT-4o - which I'm guessing is measured in the 100s of GBs. You can try out Llama 3.2 1B yourself directly in your browser (it will fetch about 1GB of da…

anyone else think 4o is kinda garbage compared to the older gpt4? as well as o1-preview and probably o1-mini. gpt4 tends to be more accurate than 4o for me.

gpt-4o is a weak version of gpt-4 with "steps-instructions". Gpt-4 is just too expensive which is why openAI is releasing all these mini versions.
Post reply on HN