Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Kimi-K3 on HuggingFace

huggingface.co

Post reply on HN
Print view View original

441–450 of 588 posts

Re: Kimi-K3 on HuggingFace

#441

Post by flockonus »

walrus01 wrote: post #50
NitpickLawyer wrote: post #3

This will be interesting for a few reasons. First, depending on where the median pricing settles w/ 3rd party providers will tell us what it costs to serve a 3T model. Since it's going to be mxfp4 native, it'll take ~1.5TB of VRAM to host this, which is juuust at the limit of 8xb200s (but realistically you'll need 16x for context / throughput optimisation). Won't be cheap to host, but at least we should get some rang…

It will be very interesting to see what kind of 'slow' performance people get from running it on a no GPU, but tons of RAM server (like a dual or quad socket xeon with 1.5 to 3TB of RAM). For the purpose of giving it longer duration tasks to generate a piece of something and come back and check on what it has done in 4 or 6 hours. Even if the output is like 5-6 tok/s, that might be usable for some purposes. Huge pric…

As you already went through the thought exercise of laying all this RAM over various slots, then match against the right CPU (which also you'll need multiple) - it becomes clear quite fast that it's trying to mimic the architecture of a GPU except in extremely low fidelity and bandwidth @ a higher energy cost.

Re: Kimi-K3 on HuggingFace

#442

Post by apitman »

nekitamo wrote: post #432
apitman wrote: post #355

Earlier quoted context omitted.

Artificial Analysis ranks qwen3.6-27b higher than qwen3.5-122b-a10b on both intelligence and coding. Does that run counter to your experience?

This tracks, in my experience the 27B is better at coding and instruction following. I'm shocked at how much of a difference the dense models vs MoE makes. But it's a moot point, because for local inference on consumer hardware, the MoE is so much faster.

Are you sure the difference is from MoE and not that 3.6 is newer?

Re: Kimi-K3 on HuggingFace

#443

Post by a11r »

gorgmah wrote: post #31

We already know that competition brought GLM 5.2 prices down roughly 45% since its release on June 16th (1.5 months ago), and the price downward slope is probably still going (I've been checking regularly and new providers keep fighting on price, I don't think prices have settled yet). For reference : https://openrouter.ai/z-ai/glm-5.2#providers I saw arguments like "Providers cannot price less than their costs" in o…

Seminanlysis is estimating sub $1 cost per MT for ~2Trillion models. The numbers change based on throughput and quant, but it is conceivable that provider costs at scale are low enough that even $2.42 per MT on GLM 5.2 (current best price) is margin positive by a wide margin.

Re: Kimi-K3 on HuggingFace

#444

Post by searealist »

rootusrootus wrote: post #431
searealist wrote: post #425

Earlier quoted context omitted.

It's easy to have your EV only charge off-peak, though. It's just a setting.

My point is that the tradeoff to get off-peak pricing is that on-peak is way, way more expensive. So you can charge the EV off-peak to maximize the savings, but everything else you do during on-peak time costs way more. Using myself as an example: I adjust my A/C to run outside of 5pm-9pm (peak) if at all possible, we try to avoid pointless high-draw usage during that same window, and both of our EVs hold off chargin…

Fair point.

I guess it depends on if you would be using ToU otherwise.

It looks like about 50% of Californians use ToU plans, but the number is only 10% nation-wide.

Re: Kimi-K3 on HuggingFace

#445

Post by fnordpiglet »

chrsw wrote: post #405
tidbeck wrote: post #401

Earlier quoted context omitted.

Comparing to Opus 5: Claude Opus 5 (Uncached Input $5/M Cached Input $0.50/M Output $25/M) but you also pay a premium on Cache write 25% for 5m and 100% for 1h.

Then there’s the questions of token efficiency and token quality.

I have to say cc opus 5 is abysmal. It talks to itself incessantly, gets stuck in minutia, fails to understand problems clearly and makes steering mistakes constantly. It also has a weird behavior where it says “ok I know exactly what to do and I will start now,” then sits waiting for user input. If you’re not on the ball you’re constantly losing 5m/1h cache. Just give me back 4.6.

Re: Kimi-K3 on HuggingFace

#446

Post by mwcampbell »

api wrote: post #375
supjeff wrote: post #369

Earlier quoted context omitted.

a "moment for humanity"? as if this shit isn't going to generate 99% slop at the cost of all we have left as a species?

Sometimes I’m not sure who is more unhinged: the total AI kool aid drinkers who think this will make us all into immortal demigods (or take over the world as it goes “foom”), or the AI doomers and haters who exaggerate everything potentially negative about it and react to it the way a 1980s Christian fundamentalist reacted to rock music. It’s a new fundamental innovation in math and CS that allows large scale lossy c…

To steelman the haters, I think their view is that the industry is so uniquely shitty that it's unconscionable to help the industry at all by using the tech, which is a product of that industry.

Re: Kimi-K3 on HuggingFace

#447

Post by nickthegreek »

m00dy wrote: post #4

There’s going to be a lot of competition around this model. Let’s see how low AI providers are willing to push prices.

They cant push it too low. The license agreement it is released under wont allow it.

> If the Licensee or any of its affiliates operates a Model as a Service business, and the aggregate revenue of the Licensee and its affiliates exceeds 20 million US dollars (or the equivalent in other currencies) in total over any consecutive 12 months, the Licensee must enter into a separate agreement with Moonshot AI before using the Software or its derivative works for any commercial purpose.

Re: Kimi-K3 on HuggingFace

#448

Post by tossandthrow »

docheinestages wrote: post #105
snemvalts wrote: post #74

Earlier quoted context omitted.

"I'd like a car that goes 300mph and gets 100mpg while doing it. I'm aware of a car that gets 100mpg but it is extremely slow." You are describing fundamental tradeoffs. Getting more performance relative to model size and training token amount is what all of the labs are solving.

Labs are focusing on creating models, small or large, that perform well on various benchmarks, including general knowledge, domain-specific expertise, and agentic capabilities. Asking for such a model while wanting to be small and fast would align with what you're describing, which I believe is different from what I'm pointing to. The model I'm describing sacrifices domain knowledge and expertise for agentic reasonin…

The core intuition here is probably that you can not separate reasoning and domain knowledge.

How are you going to use tools, if you don't have the context to use them?

Imaging if you could only think in terms of lambda calculus, and was asked to check the weather to let me know if the weekend is good for a hike.

Re: Kimi-K3 on HuggingFace

#449

Post by zozbot234 »

searealist wrote: post #428
zozbot234 wrote: post #161

Earlier quoted context omitted.

> In typical MoE model they get switched "randomly" on every token, so all experts have to be readily available. It's worse than that: a typical MoE model routes a separate set of experts at every layer , not just every token! But in practice, RAM offload (for systems with non-unified VRAM) and even SSD offload still work surprisingly well given some amount of caching. You can likely recover compute intensity and thr…

> It's worse than that: a typical MoE model routes a separate set of experts at every layer, not just every token! But in practice, RAM offload (for systems with non-unified VRAM) and even SSD offload still work surprisingly well given some amount of caching. Caching really has nothing to do with this. With RAM offload you can mostly benefit from: 1) Batching for prefill is a huge win, even with MoE, since the batch…

> With MoE it's low probability.

For even the sparsest MoE open models, having more than a handful of inferences in the batch is enough to make it more likely than not that you'll get some MoE weight reuse within any given layer. This assumes totally random sampling, ignoring any cross-request correlation that would push that probability even higher in many practical scenarios.

> With MTP it becomes _extremely_ low probability.

This is actually right, MTP is only ever worthwhile in very special cases involving either dense models or extremely wide batching of MoE ones that somehow still leaves unused room for parallelization (which AIUI would involve an assumption of very abundant compute with very limited memory bandwidth).

Re: Kimi-K3 on HuggingFace

#450

Post by [deleted] »

CamperBob2 wrote: post #408
withinboredom wrote: post #118

Earlier quoted context omitted.

We have a lossless compression codec (working on open sourcing it over the next couple of weeks) that reduces it down to its minimum entropy -- it cannot be compressed further. On all tested large models, it's a ratio of 1.34-1.23 -- and smaller models up to 3.76x. It also increases the effective bandwidth by the same rate.

We have a lossless compression codec (working on open sourcing it over the next couple of weeks) that reduces it down to its minimum entropy LOL

[deleted]
Post reply on HN