Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Kimi-K3 on HuggingFace

huggingface.co

Post reply on HN
Print view View original

281–290 of 588 posts

Re: Kimi-K3 on HuggingFace

#281

Post by [deleted] »

woadwarrior01 wrote: post #61
NitpickLawyer wrote: post #3

This will be interesting for a few reasons. First, depending on where the median pricing settles w/ 3rd party providers will tell us what it costs to serve a 3T model. Since it's going to be mxfp4 native, it'll take ~1.5TB of VRAM to host this, which is juuust at the limit of 8xb200s (but realistically you'll need 16x for context / throughput optimisation). Won't be cheap to host, but at least we should get some rang…

Since the model is natively MXFP4, I think it'll be even more interesting on the hardware front. It'll comfortably fit on a 8x AMD MI355X node. I suspect that'll drive token prices down, further.

[deleted]

Re: Kimi-K3 on HuggingFace

#282

Post by trollbridge »

vb-8448 wrote: post #153
energy123 wrote: post #111

Earlier quoted context omitted.

Gross margins are insanely important, possibly the most important single metric if for some reason you were forced to choose one.

Only if you assume that at some point, for any reason, there will be "the model" that doesn't need costly retraining. I guess this is one of the reason Anthropic i so "active" for asking for a development break.

"I want my competitors to stop competing" is an interesting ask for someone who's currently charging the highest prices in the entire industry.

Re: Kimi-K3 on HuggingFace

#283

Post by trollbridge »

Palmik wrote: post #228
NitpickLawyer wrote: post #3

This will be interesting for a few reasons. First, depending on where the median pricing settles w/ 3rd party providers will tell us what it costs to serve a 3T model. Since it's going to be mxfp4 native, it'll take ~1.5TB of VRAM to host this, which is juuust at the limit of 8xb200s (but realistically you'll need 16x for context / throughput optimisation). Won't be cheap to host, but at least we should get some rang…

You're assuming inference providers are going to sell tokens at cost. You're also assuming that the inference providers have will optimized inference engine. I haven't seen that to be the case so far, to be honest. Take a look at GLM 5 vs GLM 5.2 pricing -- GLM 5.2 cost more despite being the same model. Take a look a look at DeepSeek, which hosts DS v4, profitably, yet others aren't able or willing to match the pric…

Xioami's MiMo did match DS-V4's price, although we now know that DeepSeek set their pricing lower than they could have, due to the leaked memo, and simply decided to use "10 months to recover capex" as their yardstick. Interestingly "10 months to recover capex" is also the same price SpaceX is renting space to Anthropic and Google for.

Re: Kimi-K3 on HuggingFace

#284

Post by likium »

npn wrote: post #81
dist-epoch wrote: post #14

Earlier quoted context omitted.

> if "labs are subsidising tokens on API pricing" > SemiAnalysis estimates that Anthropic's current blended gross margin has risen to the mid-60% range, with the API business gross margin exceeding 80% Of course, people will insist "they are lying", "why should we believe them, it's well known they subsidize API pricing", ... https://newsletter.semianalysis.com/p/anthropic-3q26-profit-... https://finance.biggo.com/ne…

even deepseek, with their current (dirt cheap) price, can earn enough profit to cover the cost (hardware investment?) in 10 months.

Will the model even be competitive in 10 months though? Seems like models that reach top 20 on OpenRouter see 50% of all token spend by day 80, and 80% by day 180.

Re: Kimi-K3 on HuggingFace

#285

Post by barbacoa »

walrus01 wrote: post #50
NitpickLawyer wrote: post #3

This will be interesting for a few reasons. First, depending on where the median pricing settles w/ 3rd party providers will tell us what it costs to serve a 3T model. Since it's going to be mxfp4 native, it'll take ~1.5TB of VRAM to host this, which is juuust at the limit of 8xb200s (but realistically you'll need 16x for context / throughput optimisation). Won't be cheap to host, but at least we should get some rang…

It will be very interesting to see what kind of 'slow' performance people get from running it on a no GPU, but tons of RAM server (like a dual or quad socket xeon with 1.5 to 3TB of RAM). For the purpose of giving it longer duration tasks to generate a piece of something and come back and check on what it has done in 4 or 6 hours. Even if the output is like 5-6 tok/s, that might be usable for some purposes. Huge pric…

They are saying that AMD's new Epyc Venice CPU has 16 memory channels allowing up to 1.6Tb/s of bandwidth. Which is higher bandwidth than most non-HBM GPUs.

So full CPU local AI inference may become viable option in coming years.

Re: Kimi-K3 on HuggingFace

#286

Post by amluto »

zozbot234 wrote: post #245
amluto wrote: post #210

Earlier quoted context omitted.

Presumably it’s MoE and only needs to read a small fraction of the weights per token. Bonus points if you can get decent speculative decoding without becoming ALU-limited.

Speculative decoding is not really worthwhile for sparsely-loaded models. You end up paying in both memory bandwith and compute (loading experts based on wrongly-predicted tokens) which leaves you worse off overall. It becomes viable (even for sparse MoE) once you're batching so widely that you end up having to load most of your total weights anyway.

> Speculative decoding is not really worthwhile for sparsely-loaded models.

If wonder if you can train a model to optimize this, by trying to make the expert selection sticky across a few tokens, without too much quality loss.

Another fun idea might be to try to build a model where the router chooses the expert 1-3 tokens in advance.

Re: Kimi-K3 on HuggingFace

#287

Post by trollbridge »

walrus01 wrote: post #160
zargon wrote: post #157

Earlier quoted context omitted.

> emerging practice of using Q4 quants and Q8 KV cache for local inference That's not an emerging practice, it's a tested strategy that is these days only used as a last resort by those desperate to fit a model in memory. Some models do better than others, but generally the model quality suffers greatly under those conditions.

I have never seen anyone report "this produced really great results" from intentionally quantizing their context vs. leaving it at full precision which is the ordinary default.

Gemma's QAT is surprisingly good (although Gemma isn't that great to begin with).

Re: Kimi-K3 on HuggingFace

#288

Post by withinboredom »

xscott wrote: post #173
withinboredom wrote: post #118

Earlier quoted context omitted.

We have a lossless compression codec (working on open sourcing it over the next couple of weeks) that reduces it down to its minimum entropy -- it cannot be compressed further. On all tested large models, it's a ratio of 1.34-1.23 -- and smaller models up to 3.76x. It also increases the effective bandwidth by the same rate.

Exploring compression algorithms for weights is a good idea, and I hope you have a successful product. However, if you can prove this statement: > reduces it down to its minimum entropy -- it cannot be compressed further. I think you could make a lot more money elsewhere :-) https://en.wikipedia.org/wiki/Kolmogorov_complexity#Formal_p...

We're not an AI company ... nor do we have any reason to use it. Just a fun idea that was fruitful.

Re: Kimi-K3 on HuggingFace

#289

Post by chrsw »

Levitz wrote: post #265
chrsw wrote: post #203

Earlier quoted context omitted.

You can make the same argument for closed models. Why spend hundreds of billions training larger and larger models when you can just use Chinese models? Spend that money somewhere else further up the stack where there’s more value. Let China do the training since they’re so efficient at it.

Because without bigger players, Chinese models don't get anywhere. Playing catch-up is a radically different game.

[dead]

Re: Kimi-K3 on HuggingFace

#290

Post by trollbridge »

bavell wrote: post #234
zozbot234 wrote: post #192

Earlier quoted context omitted.

Why does it have to be so bursty though? Just let it run multiple continuous-batched inferences overnight. This would work especially well in combination with SSD offload, and given any kind of sparse attention (common in more recent models) even swapping out the KV cache itself to disk might ultimately be a win. I wouldn't be surprised if something like that ultimately became feasible for single users running even K…

I find it difficult to always have one or more long-horizon tasks 'queued up' and ready to run... I find myself usually bottlenecked on design, review, or something similar that requires me being in the driver's seat. It's possible I could queue up a bunch of tasks, letting the LLM run off in multiple directions, but then I'd be less able to steer and course correct. Just my experience though, I'm still figuring thin…

Use your favourite harness to help you find long-horizon tasks to have queued up. It's changed the structure of how my projects work a bit, and do you have to do some homework fast of how slow/fast your various providers or local inference are, but it's worth it. Start off with hobby projects so you get a feel for how it works.

I left something gargantuan running over the weekend (decompiling 1980s-era system software) and look forward to checking it out later today when I have a few free minutes.

Post reply on HN