Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Kimi-K3 on HuggingFace

huggingface.co

Post reply on HN
Print view View original

461–470 of 588 posts

Re: Kimi-K3 on HuggingFace

#461

Post by minraws »

After going through the license and trying out the model on some hardware, I don't think it will ever will be 60-70% cheaper than the price Moonshot is offering from providers, it be marginally lower sure but discounts we saw with GLM seem hard unless tps is put into the ground.

In my testing it seems like Kimi has a healthy margin (I would wager 40-50% if they are renting GPUs at full marked up prices, a bunch more otherwise, given their tps, but I don't know which GPUs they are on and what they consider margins and if they own them) but definitely not the claimed 90%+ margins of Anthropic (honestly I am suspicious of even 80% API margins for Anthropic) as I have seen some people posit. If it was just electricity costs I could bet it could be 80-90% though otherwise it seems rough given the TPS they offer.

I would love if someone has access to those super secret R100s could try it, and tell us if it's significantly cheaper since I think immediate memory optimizations seem hard since I am already on a quantised model. And not even using 1M context.

All I had access to was B200(couldn't find a B300). I am certain people could optimize it a lot better but Kimi also wants some kind of contract for big providers so I think we shouldn't imagine any significant discounts while Kimi is the top open model around.

Re: Kimi-K3 on HuggingFace

#463

Post by trollbridge »

Cider9986 wrote: post #412
skinfaxi wrote: post #386

Earlier quoted context omitted.

How do you prove you are running exclusively on Nitro enclave instances or GCP confidential spaces?

Is your local compute airgapped?

My local compute is used by me, and I'm accountable to myself whether or not is secure. So to a certain degree, I trust myself and also know what limitations / potential vulnerabilities it might have.

Re: Kimi-K3 on HuggingFace

#464

Post by DennisP »

zozbot234 wrote: post #192
schmorptron wrote: post #183

Earlier quoted context omitted.

LLM inference unfortunately also seems to be a task that's poorly formed for moderate consumer hardware,as a single user. For a single user use case, the load is bursty but requires the weights to be in memory already. So a multi user server that keeps the model weights in parts of its memory and then spends some more per user kv cache is wildly more efficient and the wildly expensive gpu cores aren't just sitting id…

Why does it have to be so bursty though? Just let it run multiple continuous-batched inferences overnight. This would work especially well in combination with SSD offload, and given any kind of sparse attention (common in more recent models) even swapping out the KV cache itself to disk might ultimately be a win. I wouldn't be surprised if something like that ultimately became feasible for single users running even K…

For software, not only that, but run batches with a cluster of agents working different parts of the same task. Software like Yegge's Gas Town has one agent act as "mayor," and others work on writing or testing various pieces, with all the agents messaging each other. In his book Yegge writes about using up to thirty agents at once.

Re: Kimi-K3 on HuggingFace

#465

Post by trollbridge »

doctorpangloss wrote: post #427
walrus01 wrote: post #64

Earlier quoted context omitted.

There are a number of use cases where sending the contents of your context and prompts (and the resulting output) to a 3rd party service is off the table as an option, and people will compromise speed for data sovereignty. And not everyone's electricity is equally expensive, I pay about $0.075 USD per kWh. It would for example cost me about $48 a month of electricity (not counting cost of cooling) to run a quad socke…

It's completely academic. At 5tok/s you can process 13 MTok per month at concurrency 1. I use 5 BILLION tokens per week when coding.

Yeah. At 5 tok/second, you're talking about around $195 worth of output tokens per month. There is no way I can run a usable K3 model for $195 a month of capex, opex, or any-kind-of-ex.

Qwen 3.6 is another matter. Paying provider rates for the amount I run locally would put me in the thousands of dollars. So that's very practical to buy a Macbook instead, plus an RTX card, and so on.

Re: Kimi-K3 on HuggingFace

#467

Post by JacobAsmuth »

NitpickLawyer wrote: post #3

This will be interesting for a few reasons. First, depending on where the median pricing settles w/ 3rd party providers will tell us what it costs to serve a 3T model. Since it's going to be mxfp4 native, it'll take ~1.5TB of VRAM to host this, which is juuust at the limit of 8xb200s (but realistically you'll need 16x for context / throughput optimisation). Won't be cheap to host, but at least we should get some rang…

It's 3/15 - https://openrouter.ai/moonshotai/kimi-k3

If you're going to open source your model, why would you set your own price high enough that other providers could easily and profitably undercut you?

Re: Kimi-K3 on HuggingFace

#469

Post by mindslight »

rootusrootus wrote: post #431
searealist wrote: post #425

Earlier quoted context omitted.

It's easy to have your EV only charge off-peak, though. It's just a setting.

My point is that the tradeoff to get off-peak pricing is that on-peak is way, way more expensive. So you can charge the EV off-peak to maximize the savings, but everything else you do during on-peak time costs way more. Using myself as an example: I adjust my A/C to run outside of 5pm-9pm (peak) if at all possible, we try to avoid pointless high-draw usage during that same window, and both of our EVs hold off chargin…

If you leave the EV charging out of your consumption, does a time of day plan still save you money on the remaining usage? Or does it cost you? If it saves you money, then it would make sense to be on a ToD plan regardless of EV charging. Which means it makes sense to consider your additional EV draw as costing the marginal off-peak rate. Essentially the EV load has the valuable property of being dispatchable.

You can do the same thought experiment with say a dehumidifier in your basement. It can easily be off during peak usage and still accomplish its job, so its cost of electricity is also the marginal off-peak rate.

Re: Kimi-K3 on HuggingFace

#470

Post by DennisP »

tossandthrow wrote: post #448
docheinestages wrote: post #105

Earlier quoted context omitted.

Labs are focusing on creating models, small or large, that perform well on various benchmarks, including general knowledge, domain-specific expertise, and agentic capabilities. Asking for such a model while wanting to be small and fast would align with what you're describing, which I believe is different from what I'm pointing to. The model I'm describing sacrifices domain knowledge and expertise for agentic reasonin…

The core intuition here is probably that you can not separate reasoning and domain knowledge. How are you going to use tools, if you don't have the context to use them? Imaging if you could only think in terms of lambda calculus, and was asked to check the weather to let me know if the weekend is good for a hike.

I think the idea here would be to not use your super smart but specialized model to check the weather. It's not obvious that it's impossible to (eg) remove most of its biology knowledge, without removing much of its ability to develop software (for example). If you're developing biology software, then don't use that particular compressed model.

(If you're claiming that it is impossible, and you have references you can share, then I'm honestly interested.)

Post reply on HN