Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Kimi-K3 on HuggingFace

huggingface.co

Post reply on HN
Print view View original

201–210 of 588 posts

Re: Kimi-K3 on HuggingFace

#201

Post by walrus01 »

cobbzilla wrote: post #187
fooker wrote: post #63

Earlier quoted context omitted.

> Even if the output is like 5-6 tok/s, that might be usable for some purposes. You'll spend ~100x more on electricity than the API cost to have it run on someone else's GPU at several hundred tokens per second. I think some sort of extreme data privacy requirement is the only situation that justifies this, but the intersection of {needs absolute data privacy, needs to run SOTA model, cannot afford GPUs} is really re…

I’ve priced it out: max $135/month to run a dual Xeon 2U server with 3T RAM & 2x 22 core Xeon Gold. It’s the 2x 750W power supplies that ultimately determine opex. My power costs $0.124/kWh, the $135 assumes drawing maximum power continuously, and in that case, I can probably offset my heating bill a little bit in the winter, so maybe effectively a little bit lower. I don’t know if that’s 100x more than I’d pay (opex…

Keep in mind that just because it has dual 750W power supplies that doesn't mean it's what its load will be, for a full CPU loaded wattage figure you'd need basically a pair of kill-a-watts plugged in inline on the feed for each poewr supply and then run stress-ng with artificial cpu stress on all cores for an hour.

Under heavy inference load you will find that the cpu usage is actually less as the bottleneck is the RAM bus speed. An older 2U rack server that is 600W load (typically a 1+1 power supply server when plugged into two kill-a-watt would show 300W on each, equal load balancing) when maxed out with stress-ng might be only 450W total running inference.

If you have 600kWh used in a month by running something 24x7 and your power is $0.15 a kWh, that's more like $90/mo (not counting cooling or any ancillary costs for the environment where it's in).

Re: Kimi-K3 on HuggingFace

#202

Post by esperent »

Jarmsy wrote: post #93
egeozcan wrote: post #51

Earlier quoted context omitted.

For me, the only format that doesn't make sense is the MM/DD/YYYY, together with its rarely seen worse sibling, MM/DD/YY (07/27/26).

Let's just go with YM/DY/MD (27/26/07)

But only for Americans, and make them weirdly dogmatic about it.

Re: Kimi-K3 on HuggingFace

#203

Post by chrsw »

teruakohatu wrote: post #143
elo02 wrote: post #136

I heard this is the talk in town these days. Why can't Meta keep up? With >10000000x more resources you'd think that they'd be able to introduce equally performant if not better open weight models

At this point in time what does meta get from releasing open weight models? Why devote the resources to it.

You can make the same argument for closed models. Why spend hundreds of billions training larger and larger models when you can just use Chinese models? Spend that money somewhere else further up the stack where there’s more value. Let China do the training since they’re so efficient at it.

Re: Kimi-K3 on HuggingFace

#204

Post by schmorptron »

zozbot234 wrote: post #192
schmorptron wrote: post #183

Earlier quoted context omitted.

LLM inference unfortunately also seems to be a task that's poorly formed for moderate consumer hardware,as a single user. For a single user use case, the load is bursty but requires the weights to be in memory already. So a multi user server that keeps the model weights in parts of its memory and then spends some more per user kv cache is wildly more efficient and the wildly expensive gpu cores aren't just sitting id…

Why does it have to be so bursty though? Just let it run multiple continuous-batched inferences overnight. This would work especially well in combination with SSD offload, and given any kind of sparse attention (common in more recent models) even swapping out the KV cache itself to disk might ultimately be a win. I wouldn't be surprised if something like that ultimately became feasible for single users running even K…

it has to be so bursty for realtime usecases like chat, which is what most people are using it for today. of course, once (if) stuff like software dark factories start working out for the average person, then you'll be able to make full use of your hardware for workload where asynchonous execution is feasible and have it run several parallel tasks overnight, with an orchestrator managing the gpu(s) allocations.

Re: Kimi-K3 on HuggingFace

#205

Post by gizmodo59 »

elo02 wrote: post #136

I heard this is the talk in town these days. Why can't Meta keep up? With >10000000x more resources you'd think that they'd be able to introduce equally performant if not better open weight models

Because lack of talent and organizational disfunction matters a lot more than you think. The reason why OAI and Ant are always at the top is because of this and I’d say compute is third on the list.

Re: Kimi-K3 on HuggingFace

#206

Post by tokai »

markasoftware wrote: post #39
gorgmah wrote: post #31

We already know that competition brought GLM 5.2 prices down roughly 45% since its release on June 16th (1.5 months ago), and the price downward slope is probably still going (I've been checking regularly and new providers keep fighting on price, I don't think prices have settled yet). For reference : https://openrouter.ai/z-ai/glm-5.2#providers I saw arguments like "Providers cannot price less than their costs" in o…

Press x to doubt on the 45% number. The cheaper providers on open router are fp4 vs fp8 for official zai. There are some cheap fp8 ones (like novita) but the ui makes it seem like it's a temporary promotion, with their normal prices being almost equal to official zai (idk much about open router so not really sure what's going on with these discounts)

There is nothing to doubt, the cheapest price on openrouter is ~45% lower than when GLM5.2 was released.

Re: Kimi-K3 on HuggingFace

#207

Post by szge »

chrsw wrote: post #203
teruakohatu wrote: post #143

Earlier quoted context omitted.

At this point in time what does meta get from releasing open weight models? Why devote the resources to it.

You can make the same argument for closed models. Why spend hundreds of billions training larger and larger models when you can just use Chinese models? Spend that money somewhere else further up the stack where there’s more value. Let China do the training since they’re so efficient at it.

as a big tech company you have the resources to make many bets and do a lot of things at the same time. it's good to have some specialists with knowledge of model training "just in case".

Re: Kimi-K3 on HuggingFace

#208

Post by mixtureoftakes »

embedding-shape wrote: post #43
davidkunz wrote: post #8

This is historic. For the first time, an open-weights LLM is right at the top. We won't be able to run this ourselves, but many providers can.

> For the first time, an open-weights LLM is right at the top. Hmm, not quite true, I think that honor, for better or worse, goes to OpenAI. When they released GPT2 (or GPT1 for that matter) is was quite literally the SOTA in the ecosystem when it was released.

thank you embedding-shape

Re: Kimi-K3 on HuggingFace

#209

Post by amluto »

frognumber wrote: post #190
light_hue_1 wrote: post #124

Earlier quoted context omitted.

As someone who has worked in two industries that are at the maximal end of data sensitivity and privacy this comes across as a tinfoil hat issue not a real business requirement. In such cases we've always found ways to trade dollars for the privacy we need without having to run our own inference at excruciating slow speeds.

Having worked in / adjacent several such industries, a lot of the question depends on scale. A trillion-dollar business can easily trade dollars for the privacy. A business with $1M to spend won't even get a phone call with OpenAI or Anthropic, who were the only* previous players in town for doing this. Worst-case example: Bootstrapped startup working in military. It's also the case that an open model enables many mo…

> Omitting Azure, which gives some privacy for some $$$ on their models, but not at the level of high-security.

If I were ranking third parties on their ability to safely handle my data without compromising it, I would rank Anthropic pretty low for things like Fable (where they more or less promise that they will misuse my data), but I want Azure pretty low in the sense that I fully expect them to be compromised.

I would tend to trust Amazon to avoid being compromised.

Re: Kimi-K3 on HuggingFace

#210

Post by amluto »

Havoc wrote: post #193
walrus01 wrote: post #50

Earlier quoted context omitted.

It will be very interesting to see what kind of 'slow' performance people get from running it on a no GPU, but tons of RAM server (like a dual or quad socket xeon with 1.5 to 3TB of RAM). For the purpose of giving it longer duration tasks to generate a piece of something and come back and check on what it has done in 4 or 6 hours. Even if the output is like 5-6 tok/s, that might be usable for some purposes. Huge pric…

> Even if the output is like 5-6 tok/s On a 3T model I’d imagine you’d be closer to 0.05 tks

Presumably it’s MoE and only needs to read a small fraction of the weights per token. Bonus points if you can get decent speculative decoding without becoming ALU-limited.
Post reply on HN