Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Kimi-K3 on HuggingFace

huggingface.co

Post reply on HN
Print view View original

291–300 of 588 posts

Re: Kimi-K3 on HuggingFace

#291

Post by serf »

swingboy wrote: post #102
NitpickLawyer wrote: post #3

This will be interesting for a few reasons. First, depending on where the median pricing settles w/ 3rd party providers will tell us what it costs to serve a 3T model. Since it's going to be mxfp4 native, it'll take ~1.5TB of VRAM to host this, which is juuust at the limit of 8xb200s (but realistically you'll need 16x for context / throughput optimisation). Won't be cheap to host, but at least we should get some rang…

Say a single Kimi K3 is deployed on 16 x B200s: how many concurrent users can that handle? I realize the question assumes a major simplification that everyone's prompts/sessions are the same.

>I realize the question assumes a major simplification that everyone's prompts/sessions are the same.

well, exactly.

that's tough to answer without just average sampling because some users will ask the model "what's todays date" or "what color is the sky?" and some users will ask "Let's rewrite the linux kernel in brainfuck."

Re: Kimi-K3 on HuggingFace

#293

Post by trollbridge »

mft_ wrote: post #129
KronisLV wrote: post #45

I feel like most hardware to run LLMs on is shaped wrong for individuals. It's either having a model struggling along with like 5-10 tokens per second on unified memory, or data center cards with hundreds of GB of VRAM consuming more than a kW of power. It doesn't seem like there's prosumer GPUs with like 180W-250W TDP and 128 GB or 256 GB of VRAM (one can dream). Then bifurcation and even just two of those cards wou…

You're right, and it's interesting to consider why. It's probably a combination of a few factors: 1) Local LLMs are a relatively new phenomenon and hardware takes years. Apple probably lucked into their unified memory architecture being suitable (in terms of memory size and bandwidth) for local LLMs, but it's only with the newest generations we're hearing about LLMs even being a consideration in their design process.…

There will be a huge market for local inference once it's cheap and widely available.

Try to imagine output token speeds of 15,000 tok/s and a time-to-first-token of 200ms. (This has already been done for Llama 8B.)

Now imagine gargantuan context windows (2M, 4M, or even bigger); keep in mind the 1M context windows were science fiction a few years ago... now imagine having this on a local model on something like a phone or portable device that can be gathering data about things you're doing and constantly run inference for things useful to you. An obvious example of this would be a chatbot you can talk to that responds like a normal human conversation and doesn't have delays, but that's just scratching the surface.

Re: Kimi-K3 on HuggingFace

#294

Post by trollbridge »

XCSme wrote: post #269
jug wrote: post #267

In my opinion, next step is to cut down on reasoning tokens while maintaining intelligence. The Chain of Thought and looping can still be an issue with these Chinese models. They in fact said K3 would improve in the area but it's still an issue that unfortunately harms the token cost wins a bit. OpenAI has been really impressive here, on the opposite end of this.

Yeah, they thing forever and doubt everything "wait but" for 200k tokens for almost any question.

On the flip side, I really like being able to inspect its reasoning chain thoroughly, as opposed to the "black box" that Anthropic models are now.

Re: Kimi-K3 on HuggingFace

#295

Post by horsawlarway »

pembrook wrote: post #225
teruakohatu wrote: post #147

Earlier quoted context omitted.

> My guess is that someone will be selling tokens at less than electricity + depreciation of GCs soon, since there's a lot of competition and "smaller" data centers have overcapacity? This is speculation, correct me if I'm wrong My guess is they are selling you the tokens, then selling your tokens (data) onto someone else.

I see these conspiratorial arguments all the time and I think people massively overestimate the value of the average users tokens. The problems with frontier models (design taste, ability to solve novel/difficult problems, etc) cannot be solved by throwing more slop from the average user at it. Actually, most of the main deficiencies in current models stem from the fact that their data sets aren’t curated and special…

I don't think the goal of this data is necessarily model improvement.

I think it's marketing, advertising, and product refinement.

Ex: all the things Google wants your search data for.

It's somewhat silly to think the value of that data has changed much. Advertisers want to know what's popular and getting clicks and attention. Competitors want to know what features are getting used in their markets.

In the simplest case, think of this data as improving the harness, not the model.

Re: Kimi-K3 on HuggingFace

#297

Post by trollbridge »

clbrmbr wrote: post #188
docheinestages wrote: post #67

Given the frontier-level capabilities of Kimi K3, I'm wondering if it's possible to extract the core capabilities (fundamental reasoning and tool calling) of the model into a smaller one that consumer devices could run? Not sure exactly how, but either by heavy distillation or some other surgical method since Kimi has a Mixture of Experts architecture. I think it's very valuable to have a smaller model that doesn't h…

There’s another way besides distillation that’s way cheaper: You can have the big model build prescriptive skills that the small model follows. Take the “train” portion of tasks on some benchmark, have K3 complete it, and then output detailed descriptions of tools used and why, then run the validation tasks with some small model that has access to the skills.

Yes. Using a harness with a strong model to create lots of utilities and tools for yourself is effectively the same thing.

Re: Kimi-K3 on HuggingFace

#299

Post by trollbridge »

davidricodias wrote: post #170
bertili wrote: post #168

Is there any (near future) technology that would permit burning this terrabyte into some kind of ROM chip?

Yes, from 6 days ago: https://news.ycombinator.com/item?id=48986351

A question of course would be "is 15,000 tok/s Gemini better than 100 tok/s Opus 5"?

Re: Kimi-K3 on HuggingFace

#300

Post by trollbridge »

walrus01 wrote: post #148
elo02 wrote: post #136

I heard this is the talk in town these days. Why can't Meta keep up? With >10000000x more resources you'd think that they'd be able to introduce equally performant if not better open weight models

The cynic in me says maybe they would be further along if they hadn't spent $80 billion on trying to build the "Metaverse" VR world. I've never met anyone who actually uses it and to the best of my knowledge it has very low mass market uptake. https://finance.yahoo.com/sectors/technology/articles/mark-z... Not exactly the best use of dollars and the labor hours of some of the best minds of our generation.

Particularly when you could vibe-code Metaverse VR for a lot less than $80bn.
Post reply on HN