Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Run Kimi K3 using 29 GB of RAM at 0.50 tok/s

github.com

Post reply on HN
Print view View original

131–140 of 181 posts

Re: Run Kimi K3 using 29 GB of RAM at 0.50 tok/s

#131

Post by IshKebab »

matheusmoreira wrote: post #111
zeryx wrote: post #67

Earlier quoted context omitted.

Rewrite the readme in my voice, use all of my comments and responses in current context as source. Don't use em dashes or other LLM things. Done, solved. Never had a problem with a readme or email since.

No idea why you're getting downvoted. Asking the AI to just copy my writing patterns dramatically reduced the amount of code comment and commit message rewriting I had to do. Ten years worth of HN comments finally became useful for something!

> Ten years worth of HN comments finally became useful for something!

This is a great idea! How do you do it practically? I assume you don't feed the whole corpus in every prompt - do you condense it somehow?

Re: Run Kimi K3 using 29 GB of RAM at 0.50 tok/s

#132

Post by ruler88 »

Really cool!!

For all of the other commenters - this project isn't about practicality today. Obviously this isn't gonna be as good as using a cloud provider. But the tool draws a line of what is possible. Combination of making the models more efficient, and making local machines more capable can one day get us to a world where very high quality local models are economically feasible.

Re: Run Kimi K3 using 29 GB of RAM at 0.50 tok/s

#133

Post by withinboredom »

chrisringrose wrote: post #94
roundup wrote: post #30

How does this project compare to https://github.com/gavamedia/deltafin ?

Hi! I'm one of the deltafin devs. The biggest difference is that this is not actually a 100% "pure" uncut Kimi K3. This is requantized to 3-bit residual, whereas deltafin is the full real unaltered k3, through and through. WASTE reads about 17 GB/token versus Deltafin’s 25.8 GB/pass—roughly. That's 34% less expert traffic, and some could argue a 34% reduction in quality.

Why quantize to int8 when k3 is int4? I could be wrong, but that’s what I remember seeing.

Re: Run Kimi K3 using 29 GB of RAM at 0.50 tok/s

#134

Post by [deleted] »

fennecbutt wrote: post #121

Idk about you guys but I'd find 0.5t/s useless. Even for long tasks. I'd rather just shell out the money to offload as much as possible to say 2x 4060ti 16gb with tensor parallelisation. Anything but that low token rate. This is the sort of thing I'd expect in 20 years for some cyberpunk esque "turtlebot" that thinks at 0.5t/s, is solar powered and performs some menial civic maintenance background task like cutting g…

[deleted]

Re: Run Kimi K3 using 29 GB of RAM at 0.50 tok/s

#135

Post by withinboredom »

throwawayffffas wrote: post #97
Catloafdev wrote: post #27

Neat! But, what do you do with a 0.5tk/s LLM? Have you tried running it via llamacpp or other software that supports naive SSD offloading to compare speeds?

You get 8 nvmes set them up in raid 0/1 across two full pcie5x16 ports and you could reach up to 4ish tokens per second, presumably.

The problem is dram bandwidth to the cpu. Each token costs roughly 20gb of traffic and ddr5 is roughly 50-80gb/s, plus you still have to run the compute sequentially. That’s your limit.

Re: Run Kimi K3 using 29 GB of RAM at 0.50 tok/s

#136

Post by yearolinuxdsktp »

marcobambini wrote: post #13
pja wrote: post #4

That README hits all my “this is authored by an LLM” instincts. I presume the codebase is also written by an LLM?

I wrote tons of software, even a programming language by hand https://github.com/marcobambini/gravity . I'm using my skills to orchestrate LLMs and agents, and I can write better code much faster. As developers, we can choose to adapt to new technologies or become extinct.

I disagree with the naysayers. The README was scannable and sectioned so I could read what I wanted. But some things I wish were clearer, and I’m not sure they would be if you wrote it by hand.

Don’t waste your time rewriting it by hand just because keyboard warriors are up in arms on HN.

The material feedback here reflects the questions who might be from someone who’s been running local LLMs, so ask the LLM to update it with that in mind.

Another poster pointed out it’s not clear what the quantization is. There’s that one paragraph but it’s confusing. What I want to know right away is: are you running the unquantized model or is it quantized? If so, how much? Use terms like Q3 or Q4 or 4-bit or 8-bit. Explain why it’s not practical to quantize less. How much precision loss do you think there is at the quantization selected? What if I had 128GB RAM and wanted to have better precision and not higher token speed—would it be a good idea to choose 4-bit instead of 3-bit for some of these layers?

Re: Run Kimi K3 using 29 GB of RAM at 0.50 tok/s

#137

Post by yearolinuxdsktp »

zeryx wrote: post #67
IshKebab wrote: post #50

Earlier quoted context omitted.

Also irrespective of the merits of LLMs, it's simply unpleasant to read LLM generated prose. It can't write well. The annoyance is compounded when you read the same poor writing everywhere . I don't know how people that shovel AI prose don't realise this. Are they not also reading other people's shitty AI text? I did see one sloperator who told his agent to copy his writing style. I have no idea if that works but it'…

Rewrite the readme in my voice, use all of my comments and responses in current context as source. Don't use em dashes or other LLM things. Done, solved. Never had a problem with a readme or email since.

The emdash allergy is shortcut sheep mindset. Emdashes are commonly accepted and orthographically correct punctuation in many languages.

Also it looks like English might not be the author’s first language—might not lead to the best training corpus.

Re: Run Kimi K3 using 29 GB of RAM at 0.50 tok/s

#138

Post by jurgenburgen »

Catloafdev wrote: post #27

Neat! But, what do you do with a 0.5tk/s LLM? Have you tried running it via llamacpp or other software that supports naive SSD offloading to compare speeds?

> Neat! But, what do you do with a 0.5tk/s LLM?

Hopefully resolve incidents faster without people pasting slop into the incident thread.

Re: Run Kimi K3 using 29 GB of RAM at 0.50 tok/s

#139

Post by fsuts »

fennecbutt wrote: post #121

Idk about you guys but I'd find 0.5t/s useless. Even for long tasks. I'd rather just shell out the money to offload as much as possible to say 2x 4060ti 16gb with tensor parallelisation. Anything but that low token rate. This is the sort of thing I'd expect in 20 years for some cyberpunk esque "turtlebot" that thinks at 0.5t/s, is solar powered and performs some menial civic maintenance background task like cutting g…

Yes it’s mostly unusable but ongoing iterations of projects like this will eventually lead to a usuable version so good to see
Post reply on HN