Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Kimi-K3 on HuggingFace

huggingface.co

Post reply on HN
Print view View original

141–150 of 588 posts

Re: Kimi-K3 on HuggingFace

#141

Post by embedding-shape »

jvuygbbkuurx wrote: post #98
embedding-shape wrote: post #68

Earlier quoted context omitted.

> If you get close in output quality, then does that matter? When you're trying to estimate/infer the costs of serving the tokens and even include the cost of training the weights in order to output tokens then yeah, why wouldn't that matter?

That only matters if you are an investor not if you are a consumer.

Well, or if you're participating in a discussion on HN where the sub-topic happens to be "if labs are subsidising tokens on API pricing" and literally the cost of serving the tokens is relevant to the sub-topic people are trying to discuss...

Re: Kimi-K3 on HuggingFace

#142

Post by ttoinou »

bjackman wrote: post #104
KronisLV wrote: post #45

I feel like most hardware to run LLMs on is shaped wrong for individuals. It's either having a model struggling along with like 5-10 tokens per second on unified memory, or data center cards with hundreds of GB of VRAM consuming more than a kW of power. It doesn't seem like there's prosumer GPUs with like 180W-250W TDP and 128 GB or 256 GB of VRAM (one can dream). Then bifurcation and even just two of those cards wou…

I agree but worth noting that it's never gonna be very practical to run LLMs like this at home. Unless we have some sort of design breakthrough, the only "sensible" way to run them is at high batch levels on shared HW. Like, yeah if I could spend a few grand on such a GPU I probably would coz I'm a rich nerd, but I'd acknowledge it as an extremely inefficient luxury, kinda like a sports car. So I think you could say…

We could make LLM inference 100x cheaper to run at home efficiently, but that solution might need to be updated every 1-2 years, whereas current GPU are useful for various others tasks and last longer

Re: Kimi-K3 on HuggingFace

#143

Post by teruakohatu »

elo02 wrote: post #136

I heard this is the talk in town these days. Why can't Meta keep up? With >10000000x more resources you'd think that they'd be able to introduce equally performant if not better open weight models

At this point in time what does meta get from releasing open weight models? Why devote the resources to it.

Re: Kimi-K3 on HuggingFace

#144

Post by ttoinou »

KronisLV wrote: post #45

I feel like most hardware to run LLMs on is shaped wrong for individuals. It's either having a model struggling along with like 5-10 tokens per second on unified memory, or data center cards with hundreds of GB of VRAM consuming more than a kW of power. It doesn't seem like there's prosumer GPUs with like 180W-250W TDP and 128 GB or 256 GB of VRAM (one can dream). Then bifurcation and even just two of those cards wou…

The next mac Ultra will allow to run a big model locally with acceptable speed. But we need people to optimize it for that computer, and we’ll be more limited in models we can choose from

128GB is enough to run a large model, quantized, REAPed, with MoE and fast SSD for model weights

Re: Kimi-K3 on HuggingFace

#145

Post by maxignol »

woctordho wrote: post #20
NitpickLawyer wrote: post #3

This will be interesting for a few reasons. First, depending on where the median pricing settles w/ 3rd party providers will tell us what it costs to serve a 3T model. Since it's going to be mxfp4 native, it'll take ~1.5TB of VRAM to host this, which is juuust at the limit of 8xb200s (but realistically you'll need 16x for context / throughput optimisation). Won't be cheap to host, but at least we should get some rang…

Speaking of finetune, currently a common practice is LoRA over bnb 4-bit base model, but I think it's time to replace bnb with GGUF as the base model format. GGUF is actively supporting new model architectures and more aggressive quantizations. I've made some proof of concept in https://github.com/woct0rdho/transformers5-qwen3.5-recipe . We can finetune Qwen3.5-35B-A3B in 16 GiB VRAM, and DeepSeek-V4-Flash (284B-A13B…

I dont quite understand why GGUF is better optimized. Are the performances better for the same amount of VRAM ?

Re: Kimi-K3 on HuggingFace

#146

Post by alexandra_au »

markasoftware wrote: post #39
gorgmah wrote: post #31

We already know that competition brought GLM 5.2 prices down roughly 45% since its release on June 16th (1.5 months ago), and the price downward slope is probably still going (I've been checking regularly and new providers keep fighting on price, I don't think prices have settled yet). For reference : https://openrouter.ai/z-ai/glm-5.2#providers I saw arguments like "Providers cannot price less than their costs" in o…

Press x to doubt on the 45% number. The cheaper providers on open router are fp4 vs fp8 for official zai. There are some cheap fp8 ones (like novita) but the ui makes it seem like it's a temporary promotion, with their normal prices being almost equal to official zai (idk much about open router so not really sure what's going on with these discounts)

If you click the provider it shows the precision, 45% off at AkashML shows FP8. The drawback is the small context window, at 96k.

Then there's 43% off at StreamLake with FP8 precision and 1M context window.

Re: Kimi-K3 on HuggingFace

#147

Post by teruakohatu »

gorgmah wrote: post #31

We already know that competition brought GLM 5.2 prices down roughly 45% since its release on June 16th (1.5 months ago), and the price downward slope is probably still going (I've been checking regularly and new providers keep fighting on price, I don't think prices have settled yet). For reference : https://openrouter.ai/z-ai/glm-5.2#providers I saw arguments like "Providers cannot price less than their costs" in o…

> My guess is that someone will be selling tokens at less than electricity + depreciation of GCs soon, since there's a lot of competition and "smaller" data centers have overcapacity? This is speculation, correct me if I'm wrong

My guess is they are selling you the tokens, then selling your tokens (data) onto someone else.

Re: Kimi-K3 on HuggingFace

#148

Post by walrus01 »

elo02 wrote: post #136

I heard this is the talk in town these days. Why can't Meta keep up? With >10000000x more resources you'd think that they'd be able to introduce equally performant if not better open weight models

The cynic in me says maybe they would be further along if they hadn't spent $80 billion on trying to build the "Metaverse" VR world. I've never met anyone who actually uses it and to the best of my knowledge it has very low mass market uptake.

https://finance.yahoo.com/sectors/technology/articles/mark-z...

Not exactly the best use of dollars and the labor hours of some of the best minds of our generation.

Re: Kimi-K3 on HuggingFace

#150

Post by NitpickLawyer »

vb-8448 wrote: post #48
NitpickLawyer wrote: post #3

This will be interesting for a few reasons. First, depending on where the median pricing settles w/ 3rd party providers will tell us what it costs to serve a 3T model. Since it's going to be mxfp4 native, it'll take ~1.5TB of VRAM to host this, which is juuust at the limit of 8xb200s (but realistically you'll need 16x for context / throughput optimisation). Won't be cheap to host, but at least we should get some rang…

> Then we'll be able to guesstimate if "labs are subsidising tokens on API pricing". No, you don't. Without training cost you can infer only the marginal cost of serving this kind of models. Moreover, you don't know the actual size of closed models (what if Fable is a 10T model? What if it's 1T?)

Training cost is directly impacted by inference cost nowadays. Most of the gains come from RL these days, and that is highly dependant on inference (~7:1 inference:training in units of compute). That's mainly because you want many roll-outs for each training scenario.

Of course inference efficiency is dictated by model architecture, size, etc. You can still guesstimate some of those and have an idea about cost/serve at several size tiers.

Post reply on HN