Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Run Kimi K3 using 29 GB of RAM at 0.50 tok/s

github.com

Post reply on HN
Print view View original

91–100 of 181 posts

Re: Run Kimi K3 using 29 GB of RAM at 0.50 tok/s

#91

Post by johnvanommen »

walrus01 wrote: post #84
mappu wrote: post #81

Standard llama.cpp can mmap the gguf, so it'll stay on disk if it doesn't fit on memory, and the kernel page cache will ensure the hot parts ("resident trunk") stay resident. What's the benefit of a custom implementation at all?

Letting the kernel use SSD based swap space for something this big would be a good way to destroy its cumulative write endurance over a period of just a couple months. I would be very interested in seeing SMART self reported drive cumulative write and wear out stats if this was done for more than a short test. In my experience llama-server is better run with --no-mmap on things that will fit entirely into RAM. Though…

> Letting the kernel use SSD based swap space for something this big would be a good way to destroy its cumulative write endurance over a period of just a couple months.

Optanes are a good option here, right?

I bought mine for $100 for each 128GB DDR4 stick. I believe write performance is off-the-charts on these, besides the fact they're c-h-e-a-p.

Re: Run Kimi K3 using 29 GB of RAM at 0.50 tok/s

#92

Post by verisimi »

zozbot234 wrote: post #42
pja wrote: post #4

That README hits all my “this is authored by an LLM” instincts. I presume the codebase is also written by an LLM?

Yup, I hate to engage in anything that looks like a "shallow dismissal" but the project documentation seems to outright contradict itself wrt. whether it's running the model at genuinely native precision (though the claimed 3-bit quant is potentially interesting) and the headline claim of achieving 2 secs/token in a mere 29GB RAM footprint looks outright nonsensical given what we know about K3 itself (~115GB in dense…

[flagged]

Re: Run Kimi K3 using 29 GB of RAM at 0.50 tok/s

#93

Post by yjftsjthsd-h »

walrus01 wrote: post #84
mappu wrote: post #81

Standard llama.cpp can mmap the gguf, so it'll stay on disk if it doesn't fit on memory, and the kernel page cache will ensure the hot parts ("resident trunk") stay resident. What's the benefit of a custom implementation at all?

Letting the kernel use SSD based swap space for something this big would be a good way to destroy its cumulative write endurance over a period of just a couple months. I would be very interested in seeing SMART self reported drive cumulative write and wear out stats if this was done for more than a short test. In my experience llama-server is better run with --no-mmap on things that will fit entirely into RAM. Though…

> Letting the kernel use SSD based swap space for something this big

Why would it swap? If stuff gets evicted from page cache is just gets reread?

Re: Run Kimi K3 using 29 GB of RAM at 0.50 tok/s

#94

Post by chrisringrose »

roundup wrote: post #30

How does this project compare to https://github.com/gavamedia/deltafin ?

Hi! I'm one of the deltafin devs. The biggest difference is that this is not actually a 100% "pure" uncut Kimi K3. This is requantized to 3-bit residual, whereas deltafin is the full real unaltered k3, through and through.

WASTE reads about 17 GB/token versus Deltafin’s 25.8 GB/pass—roughly. That's 34% less expert traffic, and some could argue a 34% reduction in quality.

Re: Run Kimi K3 using 29 GB of RAM at 0.50 tok/s

#95

Post by throwawayffffas »

bilsbie wrote: post #64

Claude might as well be .5 tok/s. I end up waiting several minutes and what it tells me could usually be summarized in under 100 words. So I could potentially live with this if it was concise.

In my experience Kimi k3 is even more verbose.

Re: Run Kimi K3 using 29 GB of RAM at 0.50 tok/s

#96

Post by forty »

freedomben wrote: post #36
gruez wrote: post #6

Earlier quoted context omitted.

>Contributors >... >claude You don't need to presume. If someone is so lazy that they tell claude to commit their code (ie. they're too lazy to run git commit themselves), the chances they reviewed the code is slim.

I often let Claude write my commit messages even when I'm the one who wrote the code. Claude is often damn good at writing commit messages, and they frequently end up much better than if I wrote it all by hand. I nearly always edit them somewhat, but it's like starting from 80% instead of 0%. Some might call it laziness, but I call it working smarter rather than harder.

A good commit message needs to explain why this change was needed/done rather than what is the content of the commit. Unless Claude also has access to the context via for example the associated issue, it simply cannot write a good commit message since it cannot generally guess why something was done from just the change. This is also the case for humans and the reason why you need to include this in the commit message.

Re: Run Kimi K3 using 29 GB of RAM at 0.50 tok/s

#97

Post by throwawayffffas »

Catloafdev wrote: post #27

Neat! But, what do you do with a 0.5tk/s LLM? Have you tried running it via llamacpp or other software that supports naive SSD offloading to compare speeds?

You get 8 nvmes set them up in raid 0/1 across two full pcie5x16 ports and you could reach up to 4ish tokens per second, presumably.
Post reply on HN