Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Kimi-K3 on HuggingFace

huggingface.co

Post reply on HN
Print view View original

421–430 of 588 posts

Re: Kimi-K3 on HuggingFace

#421

Post by entrope »

likium wrote: post #284
npn wrote: post #81

Earlier quoted context omitted.

even deepseek, with their current (dirt cheap) price, can earn enough profit to cover the cost (hardware investment?) in 10 months.

Will the model even be competitive in 10 months though? Seems like models that reach top 20 on OpenRouter see 50% of all token spend by day 80, and 80% by day 180.

As long as the hardware can be used on newer models, hardware costs can be recouped running a future model.

But if they're hoping to recoup non-recurring engineering costs rather than just hardware costs, they do need to consider the useful lifetime of the specific model.

Re: Kimi-K3 on HuggingFace

#422

Post by walrus01 »

trollbridge wrote: post #280
cobbzilla wrote: post #187

Earlier quoted context omitted.

I’ve priced it out: max $135/month to run a dual Xeon 2U server with 3T RAM & 2x 22 core Xeon Gold. It’s the 2x 750W power supplies that ultimately determine opex. My power costs $0.124/kWh, the $135 assumes drawing maximum power continuously, and in that case, I can probably offset my heating bill a little bit in the winter, so maybe effectively a little bit lower. I don’t know if that’s 100x more than I’d pay (opex…

If you actually were running this thing at 80% or 100% load, then the first thing you'd want to is get a better PDU and then connect your servers to that (48V DC).

One of the problems in buying used/refurb x86-64 rack servers for test and development/proof of concept environment, is that by volume in the market, there's not that many -48VDC power supplies going around, because maybe 5-10% of enterprise customers buy them. Resulting in many fewer units ending up on the resale market.

The options for AC power supplies for servers with 2 or 4 load sharing redundant power supplies are a lot greater. If you were buying all new hardware and starting from a clean sheet of paper design with lots of money to spend, absolutely. At that point also start looking at higher voltage DC distribution stuff related to open compute platform and 800VDC.

But if I were trying to make the absolute most use of $20,000 to put together a 3TB RAM server (48 x 64GB DIMMs), it would likely end up AC powered.

Re: Kimi-K3 on HuggingFace

#423

Post by nozzlegear »

achrono wrote: post #340
storus wrote: post #333

Getting 404 on the OP's link. Does it mean it got banned or self-censored in the meantime?

At the risk of sounding like a conspiracy theorist, this sounds like a great opportunity to make a statement . US or China, but likelier to be the former. Maybe Clem's on a call with the US government right now?

Who is Clem?

Re: Kimi-K3 on HuggingFace

#424

Post by clint »

walrus01 wrote: post #387

Looks like it's live now, it's approx 17GB per safetensors file x 96 files, so so about 1.63TB. I can only imagine that people with their favorite quantizing tools warmed up and ready to go are aggressively downloading it now.

There's a 2bit quant on HF already at ~1TB

Re: Kimi-K3 on HuggingFace

#425

Post by searealist »

rootusrootus wrote: post #415
mdasen wrote: post #309

Earlier quoted context omitted.

That's an unusually low electric rate for the US - way below the lowest state average which is Idaho at 12.4 cents. It's certainly possible that you are getting 7.5 cents including delivery, but I've had friends say that they're "getting 13 cents per kWh" here in Massachusetts, but that's just the supply rate and the delivery is another ~18 cents. There are parts of states like Grant County Washington that have cheap…

A lot of people quoting low rates are also just referring to their off-peak rate. This is pretty common in EV discussions. It's not exactly a fair argument there, either, because the flip side of having an off-peak rate is that the on-peak rate is usually quite a lot higher. So the true effective rate is a bit higher, somewhere in the middle depending on usage pattern.

It's easy to have your EV only charge off-peak, though. It's just a setting.

Re: Kimi-K3 on HuggingFace

#426

Post by midnightbobarun »

dominotw wrote: post #319
midnightbobarun wrote: post #271

K3 releasing as open source right at the same time that people are criticizing Opus for questionable performance (there's even a thread on HN about Opus 5's problems)... this is such a flex.

> (there's even a thread on HN about Opus 5's problems is opus 5 a flop like 4.8 ? where is the thread btw curios

There's two threads now, apparently

https://news.ycombinator.com/item?id=49066591

https://news.ycombinator.com/item?id=49068029

Re: Kimi-K3 on HuggingFace

#427

Post by doctorpangloss »

walrus01 wrote: post #64
fooker wrote: post #63

Earlier quoted context omitted.

> Even if the output is like 5-6 tok/s, that might be usable for some purposes. You'll spend ~100x more on electricity than the API cost to have it run on someone else's GPU at several hundred tokens per second. I think some sort of extreme data privacy requirement is the only situation that justifies this, but the intersection of {needs absolute data privacy, needs to run SOTA model, cannot afford GPUs} is really re…

There are a number of use cases where sending the contents of your context and prompts (and the resulting output) to a 3rd party service is off the table as an option, and people will compromise speed for data sovereignty. And not everyone's electricity is equally expensive, I pay about $0.075 USD per kWh. It would for example cost me about $48 a month of electricity (not counting cost of cooling) to run a quad socke…

It's completely academic. At 5tok/s you can process 13 MTok per month at concurrency 1. I use 5 BILLION tokens per week when coding.

Re: Kimi-K3 on HuggingFace

#428

Post by searealist »

zozbot234 wrote: post #161
petu wrote: post #134

Earlier quoted context omitted.

You need whole weights in VRAM for optimal performance. Don't be confused by "experts" in the name -- you don't get to load static subset of experts and blast next 100 tokens with them. In typical MoE model they get switched "randomly" on every token, so all experts have to be readily available. > With light quantization (Q6_K) that's enough to run it (slowly) on a single 5090. Kimi K2.6 is released as INT4 already.…

> In typical MoE model they get switched "randomly" on every token, so all experts have to be readily available. It's worse than that: a typical MoE model routes a separate set of experts at every layer , not just every token! But in practice, RAM offload (for systems with non-unified VRAM) and even SSD offload still work surprisingly well given some amount of caching. You can likely recover compute intensity and thr…

> It's worse than that: a typical MoE model routes a separate set of experts at every layer, not just every token! But in practice, RAM offload (for systems with non-unified VRAM) and even SSD offload still work surprisingly well given some amount of caching.

Caching really has nothing to do with this. With RAM offload you can mostly benefit from:

1) Batching for prefill is a huge win, even with MoE, since the batch sizes can be so large.

2) Keeping non-expert weights in VRAM, so the percentage of weights used per token in VRAM is higher. This benefit reduces with larger models, though.

> You can likely recover compute intensity and throughput by batching requests together, which (in practice, depending on sparsity) will end up reusing some of the loaded experts with high probability;

With MoE it's low probability.

> MTP then becomes applicable even for a MoE model

With MTP it becomes _extremely_ low probability.

Re: Kimi-K3 on HuggingFace

#429

Post by sreekanth850 »

thundergolfer wrote: post #400
sreekanth850 wrote: post #22

how feasible its will be to run on modal or deepinfra? anyone here tried and tested such large models running?

Modal eng here. Getting the model running is quite challenging, but its accessible right now on Modal via Endpoints: https://modal.com/blog/kimi-k3-by-moonshot-now-available-on-...

15/million. that will be much higher than subscribing claude or open AI right

Re: Kimi-K3 on HuggingFace

#430

Post by murderfs »

mdasen wrote: post #309
walrus01 wrote: post #64

Earlier quoted context omitted.

There are a number of use cases where sending the contents of your context and prompts (and the resulting output) to a 3rd party service is off the table as an option, and people will compromise speed for data sovereignty. And not everyone's electricity is equally expensive, I pay about $0.075 USD per kWh. It would for example cost me about $48 a month of electricity (not counting cost of cooling) to run a quad socke…

That's an unusually low electric rate for the US - way below the lowest state average which is Idaho at 12.4 cents. It's certainly possible that you are getting 7.5 cents including delivery, but I've had friends say that they're "getting 13 cents per kWh" here in Massachusetts, but that's just the supply rate and the delivery is another ~18 cents. There are parts of states like Grant County Washington that have cheap…

> Pacific Contiguous 26.1 cents

This is a bit misleading, because it's combining the 50 cents/kWh from California with 15ish cents/kWh in Oregon and Washington. Seattle City Light, for example, charges 13.38 cents/kWh on flat rate pricing, and far less with time-of-use billing (8 cents/kWh on off-peak).

Post reply on HN