Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Run Kimi K3 using 29 GB of RAM at 0.50 tok/s

github.com

Post reply on HN
Print view View original

121–130 of 181 posts

Re: Run Kimi K3 using 29 GB of RAM at 0.50 tok/s

#121

Post by fennecbutt »

Idk about you guys but I'd find 0.5t/s useless. Even for long tasks.

I'd rather just shell out the money to offload as much as possible to say 2x 4060ti 16gb with tensor parallelisation. Anything but that low token rate.

This is the sort of thing I'd expect in 20 years for some cyberpunk esque "turtlebot" that thinks at 0.5t/s, is solar powered and performs some menial civic maintenance background task like cutting grass, or scrubbing pavements. Or the "slowbot" that sits in the garden slowly pruning a bonsai, only just keeping up with the growth of the young plant.

Re: Run Kimi K3 using 29 GB of RAM at 0.50 tok/s

#122

Post by adrian_b »

walrus01 wrote: post #102
yjftsjthsd-h wrote: post #93

Earlier quoted context omitted.

> Letting the kernel use SSD based swap space for something this big Why would it swap? If stuff gets evicted from page cache is just gets reread?

The behavior from llama-server I've seen in the past is that it fills the RAM, then completely fills the swap when the GGUF won't fit in available CPU-connected + GPU RAM. I plan to do some further testing watching iostat live and other metrics for level of constant ongoing writes to the swap, to see just how detrimental it could be to SSD write life.

You should better not use swap at all, which eliminates all problems, especially on any system that has a decent amount of DRAM.

I have stopped using swap a quarter of century ago, and it was for the better.

I have seen swap advocates, but I do not agree with any of their arguments. I have encountered workloads for which the amount of memory in a computer was insufficient, so the OOM was invoked, but in all such cases I preferred to learn immediately about the existing problem and solve it by various means, e.g. increasing the amount of physical memory or reducing the amount of concurrent jobs, than to waste a lot of time because of not knowing why the performance was inadequate.

Re: Run Kimi K3 using 29 GB of RAM at 0.50 tok/s

#123

Post by Torn »

bglazer wrote: post #19
pja wrote: post #4

That README hits all my “this is authored by an LLM” instincts. I presume the codebase is also written by an LLM?

Yeah I'm begging these authors to at least *read* the LLM generated README's. They're so, so incomprehensible because the LLM has a super limited theory of mind for readers. They always assume that external readers have access to the full context and history of decisions in the project development. These decisions and instructions from the user are extremely important for the model and almost completely irrelevant fo…

On the plus side, LLM slop like this in readmes are a great indicator that you can just close the tab and carry on with life - nothing of value lost

Re: Run Kimi K3 using 29 GB of RAM at 0.50 tok/s

#124

Post by pja »

misterderpie wrote: post #37
marcobambini wrote: post #13

Earlier quoted context omitted.

I wrote tons of software, even a programming language by hand https://github.com/marcobambini/gravity . I'm using my skills to orchestrate LLMs and agents, and I can write better code much faster. As developers, we can choose to adapt to new technologies or become extinct.

> I'm using my skills to orchestrate LLMs and agents, and I can write better code much faster. The fact that the top comment on this thread calls it out, in a negative way, hints at that you aren't.

> > I'm using my skills to orchestrate LLMs and agents, and I can write better code much faster. > > The fact that the top comment on this thread calls it out, in a negative way, hints at that you aren't.

It's interesting that what I meant as a purely factual question with no prejudice either way has been taken by almost everyone to be explicitly negative & critical.

I need to be a tad less blunt I guess if I don't want to

Re: Run Kimi K3 using 29 GB of RAM at 0.50 tok/s

#125

Post by jurgenburgen »

tim-projects wrote: post #117

Once the tech catches up to the point that we can accurately select the right model for the task, then this ends up becoming a valuabke thing to have. You would spin it up sparingly as part of an automated discovery process maybe for 30 mins a day. And the rest of the time is spent using tiny models. I could see a future like that.

That’s not going to happen. We can’t even estimate how long it will take to complete a backlog item until after we complete it, there is no way to know how complex a task is without doing it.

Re: Run Kimi K3 using 29 GB of RAM at 0.50 tok/s

#126

Post by Garlef »

ikurei wrote: post #107

They say it's a waste that you pay for the tokens and then the inference provider pays for the electricity. Isn't that how everything works? I pay cucumbers and the farmers have to pay for the water and the fertilizer... I hope that reasoning is an after-the-fact justification by the LLM that wrote this. It's a ver interesting idea and I wouldn't mind trying it out, but with a smaller model. At 0.5t/s and reading man…

If you grow your own tomatoes, you'll have free tomatoes!

(Doesn't really get you a BLT but hey... at least you'll have saved the world a bit because they're not from the supermarket)

/s

Re: Run Kimi K3 using 29 GB of RAM at 0.50 tok/s

#127

Post by darkwater »

dannyw wrote: post #115
darkwater wrote: post #114

Earlier quoted context omitted.

It's not a cost if you self-consume your energy, even if you get paid for what you inject. You are just amortizing differently the cost of the PV installation. But what you get for the energy you sell is not real money, it's just money you MIGHT consume as energy in other periods.

Actually, saving $1 on something you would pay anyway is more valuable than earning $1 of taxable income. I think that's just semantics -- unless you genuinely keep racking up non-redeemable energy bill credits for feed-in.

You don't know if you are going to pay it anyway and depending on the retribution model you can loose them if you don't use them in time.

Cost is not the right mental model here, you can and should count them in the amortization plan of your installation (total cost / (projected generated kWh * lifetime)), so every kWh not used and injected is paying for it, while the kWh consumed should be discounted at the price you would pay for them at that specific moment.

> unless you genuinely keep racking up non-redeemable energy bill credits for feed-in.

I know many many cases of people with overdimensioned PV installations that at some point want to leave their current energy company because prices are changing and have hundreds or more of euros in the "virtual battery".

Re: Run Kimi K3 using 29 GB of RAM at 0.50 tok/s

#129

Post by IshKebab »

darkwater wrote: post #114
IshKebab wrote: post #110

Earlier quoted context omitted.

In the UK you can sell electricity domestically for 20c/kWh so at least here that's the cost if you have PV. If you don't, the cost is higher.

It's not a cost if you self-consume your energy, even if you get paid for what you inject. You are just amortizing differently the cost of the PV installation. But what you get for the energy you sell is not real money, it's just money you MIGHT consume as energy in other periods.

I have no idea what you're trying to say. With PV, if you consume one additional kWh your bank balance will be 20c lower than it would otherwise have been. You can argue semantics about whether that is a "cost" all you want, but practically the end result is the same and any financial decisions about whether to consume that kWh should have an identical result to what you'd decide if you bought the energy in for the same price.

Re: Run Kimi K3 using 29 GB of RAM at 0.50 tok/s

#130

Post by jnwatson »

mappu wrote: post #81

Standard llama.cpp can mmap the gguf, so it'll stay on disk if it doesn't fit on memory, and the kernel page cache will ensure the hot parts ("resident trunk") stay resident. What's the benefit of a custom implementation at all?

There was another project posted a few days ago that was quite similar and someone asked the same thing to the author. They said they tried that first and they got a 10x speed boost by doing it manually. It is the same reason why database engines write their own backing cache logic: the kernel's paging logic is generic and on-demand, while someone that's aware of the actual usage patterns can prefetch and pipeline the right data before it is used.
Post reply on HN