Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Kimi-K3 on HuggingFace

huggingface.co

Post reply on HN
Print view View original

181–190 of 588 posts

Re: Kimi-K3 on HuggingFace

#181

Post by tiagod »

criley2 wrote: post #167
vb-8448 wrote: post #48

Earlier quoted context omitted.

> Then we'll be able to guesstimate if "labs are subsidising tokens on API pricing". No, you don't. Without training cost you can infer only the marginal cost of serving this kind of models. Moreover, you don't know the actual size of closed models (what if Fable is a 10T model? What if it's 1T?)

>No, you don't. Without training cost you can infer only the marginal cost of serving this kind of models. Are you talking about Kimi's training cost or the training cost of the model(s) that Kimi distilled? Because Moonshot didn't even incur the majority of the training costs either

I believe they are talking about the closed models' training costs.

I other words, the providers that will be offering K3 inference don't have any training costs to offset, so they are only charging for the inference itself. OAI/Anthropic would need to offset their R&D and training costs in order to not be selling API access at a loss.

Re: Kimi-K3 on HuggingFace

#182

Post by Davidzheng »

ttoinou wrote: post #144
KronisLV wrote: post #45

I feel like most hardware to run LLMs on is shaped wrong for individuals. It's either having a model struggling along with like 5-10 tokens per second on unified memory, or data center cards with hundreds of GB of VRAM consuming more than a kW of power. It doesn't seem like there's prosumer GPUs with like 180W-250W TDP and 128 GB or 256 GB of VRAM (one can dream). Then bifurcation and even just two of those cards wou…

The next mac Ultra will allow to run a big model locally with acceptable speed. But we need people to optimize it for that computer, and we’ll be more limited in models we can choose from 128GB is enough to run a large model, quantized, REAPed, with MoE and fast SSD for model weights

Not Kimi K3 large though

Re: Kimi-K3 on HuggingFace

#183

Post by schmorptron »

KronisLV wrote: post #45

I feel like most hardware to run LLMs on is shaped wrong for individuals. It's either having a model struggling along with like 5-10 tokens per second on unified memory, or data center cards with hundreds of GB of VRAM consuming more than a kW of power. It doesn't seem like there's prosumer GPUs with like 180W-250W TDP and 128 GB or 256 GB of VRAM (one can dream). Then bifurcation and even just two of those cards wou…

LLM inference unfortunately also seems to be a task that's poorly formed for moderate consumer hardware,as a single user. For a single user use case, the load is bursty but requires the weights to be in memory already. So a multi user server that keeps the model weights in parts of its memory and then spends some more per user kv cache is wildly more efficient and the wildly expensive gpu cores aren't just sitting idle most of the time. Don't get me wrong, most desktop workloads are bursty, but the power needed to to them has gotten cheap enough that we can have way overkill for idle scenarios hardware just sitting on our desks.

Re: Kimi-K3 on HuggingFace

#184

Post by clbrmbr »

TurdF3rguson wrote: post #169

FYI huggingface refers to the alien from the Aliens movies that we need to prevent from reaching earth at any cost because it means the end of civilization. Just checking in because y'all sound good with that.

Funny retcon, but come on… TIL huggingface started as a chatbot for teens who didn’t get enuf hugs.

Re: Kimi-K3 on HuggingFace

#185

Post by zozbot234 »

walrus01 wrote: post #85
magicalhippo wrote: post #80

Earlier quoted context omitted.

> running it on a no GPU, but tons of RAM server Or from SSD using something like Colibri[1]. Not going to be quick, but at least runable. [1]: https://github.com/JustVugg/colibri

It's a great concept but I think it would cross the line from 'very slow' to 'so slow it's unusable' at this size. Even if we say you have an NVME SSD that does 7GB/s reads, that's dramatically slower than being able to hold the whole thing in DRAM. Like the difference between 1.3 tok/s in RAM vs 0.1 tok/s with a colibri-like method. edit: the results I have seen from people trying colibri with fast consumer grade PC…

For single stream inference of a MoE model, the size of active sparse parameters will matter a lot more than total parameters. This is generally around half of the reported active parameter count - the other half being a dense subset that can be easily cached in VRAM even on fairly modest consumer setups. So the achievable performance may be quite a bit better than a naïve assessment might suggest.

Re: Kimi-K3 on HuggingFace

#186

Post by pmdr »

someguyornotidk wrote: post #54
pmg1991 wrote: post #30

Hoping no issues on Huggingface due to download rush.

For huge models like these, the only reasonable way to host them is via torrents. I don't understand why hf doesn't offer this as an option. Linux distributions got this right: Offer both HTTP and Torrents. Let the user decide.

> Let the user decide.

Perhaps that's exactly what they're trying to avoid, giving the user any form of control and having them depend on HF.

Re: Kimi-K3 on HuggingFace

#187

Post by cobbzilla »

fooker wrote: post #63
walrus01 wrote: post #50

Earlier quoted context omitted.

It will be very interesting to see what kind of 'slow' performance people get from running it on a no GPU, but tons of RAM server (like a dual or quad socket xeon with 1.5 to 3TB of RAM). For the purpose of giving it longer duration tasks to generate a piece of something and come back and check on what it has done in 4 or 6 hours. Even if the output is like 5-6 tok/s, that might be usable for some purposes. Huge pric…

> Even if the output is like 5-6 tok/s, that might be usable for some purposes. You'll spend ~100x more on electricity than the API cost to have it run on someone else's GPU at several hundred tokens per second. I think some sort of extreme data privacy requirement is the only situation that justifies this, but the intersection of {needs absolute data privacy, needs to run SOTA model, cannot afford GPUs} is really re…

I’ve priced it out: max $135/month to run a dual Xeon 2U server with 3T RAM & 2x 22 core Xeon Gold. It’s the 2x 750W power supplies that ultimately determine opex. My power costs $0.124/kWh, the $135 assumes drawing maximum power continuously, and in that case, I can probably offset my heating bill a little bit in the winter, so maybe effectively a little bit lower.

I don’t know if that’s 100x more than I’d pay (opex-wise) with an nvidia setup, but I can say the one-time capex is a great deal cheaper. Avoiding VRAM and DDR5 (fast DDR4 should be OK) are the biggest cost savers. ECC RAM is worth the extra price. General datacenter-quality hardware has less price sensitivity, and plenty of bang for your buck.

Re: Kimi-K3 on HuggingFace

#188

Post by clbrmbr »

docheinestages wrote: post #67

Given the frontier-level capabilities of Kimi K3, I'm wondering if it's possible to extract the core capabilities (fundamental reasoning and tool calling) of the model into a smaller one that consumer devices could run? Not sure exactly how, but either by heavy distillation or some other surgical method since Kimi has a Mixture of Experts architecture. I think it's very valuable to have a smaller model that doesn't h…

There’s another way besides distillation that’s way cheaper: You can have the big model build prescriptive skills that the small model follows.

Take the “train” portion of tasks on some benchmark, have K3 complete it, and then output detailed descriptions of tools used and why, then run the validation tasks with some small model that has access to the skills.

Re: Kimi-K3 on HuggingFace

#189

Post by LtdJorge »

walrus01 wrote: post #85
magicalhippo wrote: post #80

Earlier quoted context omitted.

> running it on a no GPU, but tons of RAM server Or from SSD using something like Colibri[1]. Not going to be quick, but at least runable. [1]: https://github.com/JustVugg/colibri

It's a great concept but I think it would cross the line from 'very slow' to 'so slow it's unusable' at this size. Even if we say you have an NVME SSD that does 7GB/s reads, that's dramatically slower than being able to hold the whole thing in DRAM. Like the difference between 1.3 tok/s in RAM vs 0.1 tok/s with a colibri-like method. edit: the results I have seen from people trying colibri with fast consumer grade PC…

On a server machine you can have more than 100GB/s of NVMe if you parallelize (RAID 0 and the like). But it's still gonna be noticeably slower.

Re: Kimi-K3 on HuggingFace

#190

Post by frognumber »

light_hue_1 wrote: post #124
walrus01 wrote: post #64

Earlier quoted context omitted.

There are a number of use cases where sending the contents of your context and prompts (and the resulting output) to a 3rd party service is off the table as an option, and people will compromise speed for data sovereignty. And not everyone's electricity is equally expensive, I pay about $0.075 USD per kWh. It would for example cost me about $48 a month of electricity (not counting cost of cooling) to run a quad socke…

As someone who has worked in two industries that are at the maximal end of data sensitivity and privacy this comes across as a tinfoil hat issue not a real business requirement. In such cases we've always found ways to trade dollars for the privacy we need without having to run our own inference at excruciating slow speeds.

Having worked in / adjacent several such industries, a lot of the question depends on scale.

A trillion-dollar business can easily trade dollars for the privacy. A business with $1M to spend won't even get a phone call with OpenAI or Anthropic, who were the only* previous players in town for doing this.

Worst-case example: Bootstrapped startup working in military.

It's also the case that an open model enables many more intermediate-cost solutions. E.g. providers certified for specific applications, on-prem rentals, etc.

* Omitting Azure, which gives some privacy for some $$$ on their models, but not at the level of high-security.

Post reply on HN