Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Run Kimi K3 using 29 GB of RAM at 0.50 tok/s

github.com

Post reply on HN
Print view View original

101–110 of 181 posts

Re: Run Kimi K3 using 29 GB of RAM at 0.50 tok/s

#101

Post by walrus01 »

mappu wrote: post #90
walrus01 wrote: post #84

Earlier quoted context omitted.

Letting the kernel use SSD based swap space for something this big would be a good way to destroy its cumulative write endurance over a period of just a couple months. I would be very interested in seeing SMART self reported drive cumulative write and wear out stats if this was done for more than a short test. In my experience llama-server is better run with --no-mmap on things that will fit entirely into RAM. Though…

The really big thing is the model weights, which are a read workload not a write one, it won't affect an SSD's write endurance. The write workloads are just the context and any K/V cache - llama.cpp does not mmap those to disk, so they would remain in memory or VRAM as space affords.

I plan to give it a try in a day or two with llama-server from the main branch compiled today, when my Q8 GGUF download of K3 finishes, on a system with 256GB (should be more than ample for context and KV cache and a moderate chunk of the whole 1.6TB). If it works it's going to be sloooooooow as hell, but it'll be an interesting data point to see just how slow.

Re: Run Kimi K3 using 29 GB of RAM at 0.50 tok/s

#102

Post by walrus01 »

yjftsjthsd-h wrote: post #93
walrus01 wrote: post #84

Earlier quoted context omitted.

Letting the kernel use SSD based swap space for something this big would be a good way to destroy its cumulative write endurance over a period of just a couple months. I would be very interested in seeing SMART self reported drive cumulative write and wear out stats if this was done for more than a short test. In my experience llama-server is better run with --no-mmap on things that will fit entirely into RAM. Though…

> Letting the kernel use SSD based swap space for something this big Why would it swap? If stuff gets evicted from page cache is just gets reread?

The behavior from llama-server I've seen in the past is that it fills the RAM, then completely fills the swap when the GGUF won't fit in available CPU-connected + GPU RAM. I plan to do some further testing watching iostat live and other metrics for level of constant ongoing writes to the swap, to see just how detrimental it could be to SSD write life.

Re: Run Kimi K3 using 29 GB of RAM at 0.50 tok/s

#103

Post by marcobambini »

airstrafer wrote: post #73
orliesaurus wrote: post #72

Earlier quoted context omitted.

no, this is sqliteai aka SQLite Cloud, Inc

wow. shamelessly stealing the name of one of the most robust and effective libraries ever, to push some ai bullshit. what an embarrassment

We are backed by the SQLite author, and we have the right to use the SQLite name. Please do your homework before writing such comments.

We used an Elastic license for some projects to protect our work from being used in SASS without a prior agreement. WASTE is and will always be available with a very permissive license.

I really don't understand why some people prefer to spread hate instead of just asking for clarification first.

Re: Run Kimi K3 using 29 GB of RAM at 0.50 tok/s

#106

Post by cdud3 »

pja wrote: post #4

That README hits all my “this is authored by an LLM” instincts. I presume the codebase is also written by an LLM?

I prefer to have such detailed readme's created by a LLM while iterating over no iteration documentation at all and usually the later is the standard.

Re: Run Kimi K3 using 29 GB of RAM at 0.50 tok/s

#107

Post by ikurei »

They say it's a waste that you pay for the tokens and then the inference provider pays for the electricity. Isn't that how everything works? I pay cucumbers and the farmers have to pay for the water and the fertilizer...

I hope that reasoning is an after-the-fact justification by the LLM that wrote this.

It's a ver interesting idea and I wouldn't mind trying it out, but with a smaller model. At 0.5t/s and reading many gigabytes from the SDD every second... I wonder if this wouldn't be extremely practical if targeting a 500gib or 250gib model, something that is still outside most consumers' laptop.

Re: Run Kimi K3 using 29 GB of RAM at 0.50 tok/s

#110

Post by IshKebab »

darkwater wrote: post #104
bgirard wrote: post #29

Approximate calculation is putting the cost at ~$5 per million tokens (assuming 42W sustained, 20¢/kWh), and that's excluding hardware and other costs.

And what if I have PV?

In the UK you can sell electricity domestically for 20c/kWh so at least here that's the cost if you have PV. If you don't, the cost is higher.
Post reply on HN