Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Run Kimi K3 using 29 GB of RAM at 0.50 tok/s

github.com

Post reply on HN
Print view View original

171–180 of 181 posts

Re: Run Kimi K3 using 29 GB of RAM at 0.50 tok/s

#171

Post by danw1979 »

darkwater wrote: post #114
IshKebab wrote: post #110

Earlier quoted context omitted.

In the UK you can sell electricity domestically for 20c/kWh so at least here that's the cost if you have PV. If you don't, the cost is higher.

It's not a cost if you self-consume your energy, even if you get paid for what you inject. You are just amortizing differently the cost of the PV installation. But what you get for the energy you sell is not real money, it's just money you MIGHT consume as energy in other periods.

I think this is the contentious point in this thread: in the UK you can get actual cash for exporting energy to the grid, at similar rates to import, with no strings attached.

Is the model you're describing (usage credits) in the European market somewhere ?

Re: Run Kimi K3 using 29 GB of RAM at 0.50 tok/s

#172

Post by fragmede »

root_axis wrote: post #170
preommr wrote: post #147

Earlier quoted context omitted.

I pay $20 for codex, use it daily for coding, and still haven't dipped below 50% for weekly usage. I wouldn't even be able to afford buying 29 gigs of ram, or a new video card with hardware prices the way they are now. Maybe if it was 2016-2018 prices, I'd think about it.

Well, if you only ever needed a $20 subscription's worth of tokens it would never make any sense to buy hardware since even at 2018 prices you could buy 20 years of subscriptions.

Assuming $20 subscriptions continue to be available for 20 years. Maybe they will, I don't have a time machine, but $20 subscriptions seem to be the $1 Uber ride level of pricing.

Re: Run Kimi K3 using 29 GB of RAM at 0.50 tok/s

#173

Post by nijave »

darkwater wrote: post #114
IshKebab wrote: post #110

Earlier quoted context omitted.

In the UK you can sell electricity domestically for 20c/kWh so at least here that's the cost if you have PV. If you don't, the cost is higher.

It's not a cost if you self-consume your energy, even if you get paid for what you inject. You are just amortizing differently the cost of the PV installation. But what you get for the energy you sell is not real money, it's just money you MIGHT consume as energy in other periods.

Opportunity cost

Re: Run Kimi K3 using 29 GB of RAM at 0.50 tok/s

#174

Post by Wowfunhappy »

danw1979 wrote: post #169
Lerc wrote: post #167

Earlier quoted context omitted.

It's about a million tokens a week. So you can do the usual calculations of rent Vs buy. There are definitely some tasks, that if the system can run unsupervised (a largeish if), it doesn't matter as long as the result happens before a deadline. In that respect it is easy to tell if this works for you or not. As a stepping stone to more efficiency in the future it has more value.

That’s a good way to think about it - do you want your expensive laptop grinding away constantly 24/7 consuming $5 of power to generate what amounts to $15 worth of tokens per week ? You can get a frontier model subscription for about the same cost as the electricity (heavily subsidised by someone else’s money !) with instant results. I know there are applications for this and it’s cool people are pushing the boundar…

Would it really be $5 of power? I know it's hard to say definitely but even over a week that seems high to me for Apple Silicon. Don't those processors sip power?

Probably a useless comparison, but in my NYC studio apartment, my electric bill is usually ~$75 per week during the summer. That includes air conditioning, appliances, etc etc, not to mention the Haswell Desktop PC I keep running 24/7, which is rarely idle because I queue up compile jobs and stuff.

Re: Run Kimi K3 using 29 GB of RAM at 0.50 tok/s

#175

Post by root_axis »

fragmede wrote: post #172
root_axis wrote: post #170

Earlier quoted context omitted.

Well, if you only ever needed a $20 subscription's worth of tokens it would never make any sense to buy hardware since even at 2018 prices you could buy 20 years of subscriptions.

Assuming $20 subscriptions continue to be available for 20 years. Maybe they will, I don't have a time machine, but $20 subscriptions seem to be the $1 Uber ride level of pricing.

Sure, I'm just saying, even for "mundane" computing tasks where the hardware is cheap, if you're optimizing for cost, renting hardware is always the better choice.

Re: Run Kimi K3 using 29 GB of RAM at 0.50 tok/s

#176

Post by darkoob12 »

One thing about local llm projects is that they don't mention the maximum context length which for this project is 4k. I think for it to be used with a harness you need at least 200k and reliable tool calling and structured output which obviously degrade at high quantizations.

Re: Run Kimi K3 using 29 GB of RAM at 0.50 tok/s

#178

Post by darkwater »

IshKebab wrote: post #163
knollimar wrote: post #150

Earlier quoted context omitted.

The grid doesn't give you money in these locations. They give you credits for future consumption. You can only spend this on energy. If you always have a surplus, you will just rack up large amounts of credits.

What locations? In the UK they give you actual money. You can withdraw it.

In Spain you cannit withdraw it, only spend it with the same provider. If you migrate it, you lose it. And it makes sense because it's for self-consumption, you are not an energy micro-provider.

Re: Run Kimi K3 using 29 GB of RAM at 0.50 tok/s

#179

Post by ElectricalUnion »

Dylan16807 wrote: post #70
ElectricalUnion wrote: post #54

Earlier quoted context omitted.

Then have it summarise the meeting transcription over the entire week for the report next week . ftfy.

If we estimate a meeting with pauses between speakers as 2.25 words per second, and .75 words per token, then a meeting generates 3 tokens per second. This says prefill and decode are both .5 tokens per second? Then each hour of meeting turns into 6 hours to read and 1 hour to output a summary. You could summarize two hours of meeting overnight, not too bad.

Aren't those speeds for the first few tokens, that, because of no context for attention to attend, are much faster to compute that the others? I expect the actual token speed to nosedive sharply as you get more context utilization.

What about the system prompt you're using? General purpose harness like Claude Code will insert their happy 22k tokens, even before your first useful token is processed. That might make it a no-go even _before_ you can even start, as the maximum context for this seems pretty limited (to make it fast)

And those LLMs are all "thinking", that is, rather that "one-shoting" the answer, they generate a lot of internal use reasoning tokens before starting to generate useful, visible response tokens. You can easily get to 30k tokens when your initial prompt is vague ambiguous garbage (as are naive transcriptions) as your LLM will "But wait, the user might have meant X, let me think more about this" lots of times.

No thinking (therefore much worse answers) will be a requirement.

Re: Run Kimi K3 using 29 GB of RAM at 0.50 tok/s

#180

Post by Dylan16807 »

ElectricalUnion wrote: post #179
Dylan16807 wrote: post #70

Earlier quoted context omitted.

If we estimate a meeting with pauses between speakers as 2.25 words per second, and .75 words per token, then a meeting generates 3 tokens per second. This says prefill and decode are both .5 tokens per second? Then each hour of meeting turns into 6 hours to read and 1 hour to output a summary. You could summarize two hours of meeting overnight, not too bad.

Aren't those speeds for the first few tokens, that, because of no context for attention to attend, are much faster to compute that the others? I expect the actual token speed to nosedive sharply as you get more context utilization. What about the system prompt you're using? General purpose harness like Claude Code will insert their happy 22k tokens, even before your first useful token is processed. That might make it…

I have no idea how this thing performs over time. But you don't need to give it a fancy prompt to summarize, and you could cache the prompted state too.

Yes I'm assuming you one-shot the summary or close to it. A summary doesn't need a zillion tokens of extra processing. I doubt it will be "much worse" for this use case, but I admit I don't know how low you can turn the thinking.

Post reply on HN