Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Kimi-K3 on HuggingFace

huggingface.co

Post reply on HN
Print view View original

121–130 of 588 posts

Re: Kimi-K3 on HuggingFace

#121

Post by vb-8448 »

amelius wrote: post #52
vb-8448 wrote: post #48

Earlier quoted context omitted.

> Then we'll be able to guesstimate if "labs are subsidising tokens on API pricing". No, you don't. Without training cost you can infer only the marginal cost of serving this kind of models. Moreover, you don't know the actual size of closed models (what if Fable is a 10T model? What if it's 1T?)

> Without training cost you can infer only the marginal cost of serving this kind of models. Which is by far the most interesting number of the two. > Moreover, you don't know the actual size of closed models (what if Fable is a 10T model? What if it's 1T?) If you get close in output quality, then does that matter?

> Which is by far the most interesting number of the two.

Only if you don't have to continuously train new models, and you are not at a runway risk.

Re: Kimi-K3 on HuggingFace

#122

Post by magicalhippo »

walrus01 wrote: post #85
magicalhippo wrote: post #80

Earlier quoted context omitted.

> running it on a no GPU, but tons of RAM server Or from SSD using something like Colibri[1]. Not going to be quick, but at least runable. [1]: https://github.com/JustVugg/colibri

It's a great concept but I think it would cross the line from 'very slow' to 'so slow it's unusable' at this size. Even if we say you have an NVME SSD that does 7GB/s reads, that's dramatically slower than being able to hold the whole thing in DRAM. Like the difference between 1.3 tok/s in RAM vs 0.1 tok/s with a colibri-like method. edit: the results I have seen from people trying colibri with fast consumer grade PC…

It claims to support using multiple devices RAID-0 style, which should boost performance, but yea probably not very useful for most.

But still fun you can run it at home.

Re: Kimi-K3 on HuggingFace

#123

Post by kaxu-64 »

withinboredom wrote: post #118
NitpickLawyer wrote: post #3

This will be interesting for a few reasons. First, depending on where the median pricing settles w/ 3rd party providers will tell us what it costs to serve a 3T model. Since it's going to be mxfp4 native, it'll take ~1.5TB of VRAM to host this, which is juuust at the limit of 8xb200s (but realistically you'll need 16x for context / throughput optimisation). Won't be cheap to host, but at least we should get some rang…

We have a lossless compression codec (working on open sourcing it over the next couple of weeks) that reduces it down to its minimum entropy -- it cannot be compressed further. On all tested large models, it's a ratio of 1.34-1.23 -- and smaller models up to 3.76x. It also increases the effective bandwidth by the same rate.

[flagged]

Re: Kimi-K3 on HuggingFace

#124

Post by light_hue_1 »

walrus01 wrote: post #64
fooker wrote: post #63

Earlier quoted context omitted.

> Even if the output is like 5-6 tok/s, that might be usable for some purposes. You'll spend ~100x more on electricity than the API cost to have it run on someone else's GPU at several hundred tokens per second. I think some sort of extreme data privacy requirement is the only situation that justifies this, but the intersection of {needs absolute data privacy, needs to run SOTA model, cannot afford GPUs} is really re…

There are a number of use cases where sending the contents of your context and prompts (and the resulting output) to a 3rd party service is off the table as an option, and people will compromise speed for data sovereignty. And not everyone's electricity is equally expensive, I pay about $0.075 USD per kWh. It would for example cost me about $48 a month of electricity (not counting cost of cooling) to run a quad socke…

As someone who has worked in two industries that are at the maximal end of data sensitivity and privacy this comes across as a tinfoil hat issue not a real business requirement. In such cases we've always found ways to trade dollars for the privacy we need without having to run our own inference at excruciating slow speeds.

Re: Kimi-K3 on HuggingFace

#125

Post by blks »

NitpickLawyer wrote: post #3

This will be interesting for a few reasons. First, depending on where the median pricing settles w/ 3rd party providers will tell us what it costs to serve a 3T model. Since it's going to be mxfp4 native, it'll take ~1.5TB of VRAM to host this, which is juuust at the limit of 8xb200s (but realistically you'll need 16x for context / throughput optimisation). Won't be cheap to host, but at least we should get some rang…

If it takes so much resource to run, how does the sharing of a single llm works? There is some interface that basically submits context/cache plus current promt, from each user, doing basically time-sharing compute?

Re: Kimi-K3 on HuggingFace

#126

Post by NitpickLawyer »

vorticalbox wrote: post #100
NitpickLawyer wrote: post #3

This will be interesting for a few reasons. First, depending on where the median pricing settles w/ 3rd party providers will tell us what it costs to serve a 3T model. Since it's going to be mxfp4 native, it'll take ~1.5TB of VRAM to host this, which is juuust at the limit of 8xb200s (but realistically you'll need 16x for context / throughput optimisation). Won't be cheap to host, but at least we should get some rang…

I think cursor will likely do a grok fine tune rather than a kimi one for the next composer. they noted in their blog post they didn't focus purely on coding for grok 4.5.

Most likely, since they were acquired. But for us outsiders it would be a cool thing, to see if the delta is the same between kimi2.x + cursor data -> kimi3 + cursor data.

Re: Kimi-K3 on HuggingFace

#127

Post by withinboredom »

kaxu-64 wrote: post #123
withinboredom wrote: post #118

Earlier quoted context omitted.

We have a lossless compression codec (working on open sourcing it over the next couple of weeks) that reduces it down to its minimum entropy -- it cannot be compressed further. On all tested large models, it's a ratio of 1.34-1.23 -- and smaller models up to 3.76x. It also increases the effective bandwidth by the same rate.

[flagged]

Thanks for your opinion

Re: Kimi-K3 on HuggingFace

#128

Post by walrus01 »

light_hue_1 wrote: post #124
walrus01 wrote: post #64

Earlier quoted context omitted.

There are a number of use cases where sending the contents of your context and prompts (and the resulting output) to a 3rd party service is off the table as an option, and people will compromise speed for data sovereignty. And not everyone's electricity is equally expensive, I pay about $0.075 USD per kWh. It would for example cost me about $48 a month of electricity (not counting cost of cooling) to run a quad socke…

As someone who has worked in two industries that are at the maximal end of data sensitivity and privacy this comes across as a tinfoil hat issue not a real business requirement. In such cases we've always found ways to trade dollars for the privacy we need without having to run our own inference at excruciating slow speeds.

Do you mean by trading dollars for the privacy you need as:

a) Contracting with a third-party independent inference provider who will run your choice of model on fast hardware that they own, with all appropriate data security/privacy/contractual/compliance protection in place

or

b) Contracting with the original creators of the model to run inference via their API and with assurances that all the same data protection is in place

or

c) Spending the money to buy your own inference hardware to run it on something you fully own/control at proper usable speeds?

Edit: Everything I've been writing in this thread is mostly within the context of being able to evaluate K3 and its usefulness to be self-hosted as a preliminary proof of concept or test of feasibility of a new thing, such as on <$20,000 of server hardware, before proceeding to spend 300-400k on GPU-related hardware, or external third party services/ongoing billing.

Re: Kimi-K3 on HuggingFace

#129

Post by mft_ »

KronisLV wrote: post #45

I feel like most hardware to run LLMs on is shaped wrong for individuals. It's either having a model struggling along with like 5-10 tokens per second on unified memory, or data center cards with hundreds of GB of VRAM consuming more than a kW of power. It doesn't seem like there's prosumer GPUs with like 180W-250W TDP and 128 GB or 256 GB of VRAM (one can dream). Then bifurcation and even just two of those cards wou…

You're right, and it's interesting to consider why. It's probably a combination of a few factors:

1) Local LLMs are a relatively new phenomenon and hardware takes years. Apple probably lucked into their unified memory architecture being suitable (in terms of memory size and bandwidth) for local LLMs, but it's only with the newest generations we're hearing about LLMs even being a consideration in their design process.

2) NVidia seem to be deliberately blocking consumers from taking this path - as evidenced by the removal of NVLink from the 30x0 series onwards - probably to protect their data center cards from internal competition?

3) Perhaps there's just not the market for it? It's feasible that the number of nerds interested local LLMs is very small in numbers, sales, and profit potential compared to gamers on the one side, and data centers on the other. (This would explain why AMD and Intel aren't trying to out-innovate NVidia in this area, despite it being an obvious opportunity.)

Re: Kimi-K3 on HuggingFace

#130

Post by m00dy »

user43928 wrote: post #76
torginus wrote: post #17

Earlier quoted context omitted.

I think the results might be underwhelming - AI providers need to turn a profit and can't subsidize, and they're working off of the commodity hardware everyone does. I wouldn't be surprised if they started offering potentiall bad quantizations with much reduced capability at lower prices (without telling the users, of course)

I would be surprised, considering that OpenRouter requires disclosing the quantization and shows automatic benchmarks to compare between providers for the same model.

The latter is a joke
Post reply on HN