Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Kimi-K3 on HuggingFace

huggingface.co

Post reply on HN
Print view View original

71–80 of 588 posts

Re: Kimi-K3 on HuggingFace

#71

Post by embedding-shape »

walrus01 wrote: post #55
maelito wrote: post #12

Did someone run censorship and political bias tests on this ? Must be interesting.

As a completely one person, single sample anecdote, the 'heretic' uncensored Q8 GGUF variants several people have published of Qwen 3.5-122, 3.6-27B and 3.6-35B-A3B will very happily discuss just about any controversial topic that the CCP hates. Including lots of things that would get you thrown into prison if you published them in Mandarin on the domestic Chinese internet. https://github.com/p-e-w/heretic As a side…

That's a lot of words to say "No, no have has seemingly done that yet with K3".

Re: Kimi-K3 on HuggingFace

#72

Post by embedding-shape »

psychoslave wrote: post #60
ra wrote: post #6

Earlier quoted context omitted.

or 27/7 for the rest of the world

Most life forms don’t use any calendar actually.

Don't even have to go that far, outside of white-collar jobs and some groups weirdly obsessed with scheduling, most people don't use calendars at all, but their manager/boss/significant-other does that for them :)

Re: Kimi-K3 on HuggingFace

#73

Post by walrus01 »

embedding-shape wrote: post #71
walrus01 wrote: post #55

Earlier quoted context omitted.

As a completely one person, single sample anecdote, the 'heretic' uncensored Q8 GGUF variants several people have published of Qwen 3.5-122, 3.6-27B and 3.6-35B-A3B will very happily discuss just about any controversial topic that the CCP hates. Including lots of things that would get you thrown into prison if you published them in Mandarin on the domestic Chinese internet. https://github.com/p-e-w/heretic As a side…

That's a lot of words to say "No, no have has seemingly done that yet with K3".

Indeed not, but I was saying there's more than ample precedent which is tested/working and actually doesn't refuse anything. Go to the Huggingface 'models' search interface and type in "heretic". Or uncensored. One example would be: https://huggingface.co/HauhauCS/Qwen3.5-122B-A10B-Uncensored...

Re: Kimi-K3 on HuggingFace

#74

Post by snemvalts »

docheinestages wrote: post #67

Given the frontier-level capabilities of Kimi K3, I'm wondering if it's possible to extract the core capabilities (fundamental reasoning and tool calling) of the model into a smaller one that consumer devices could run? Not sure exactly how, but either by heavy distillation or some other surgical method since Kimi has a Mixture of Experts architecture. I think it's very valuable to have a smaller model that doesn't h…

"I'd like a car that goes 300mph and gets 100mpg while doing it. I'm aware of a car that gets 100mpg but it is extremely slow."

You are describing fundamental tradeoffs. Getting more performance relative to model size and training token amount is what all of the labs are solving.

Re: Kimi-K3 on HuggingFace

#75

Post by fooker »

walrus01 wrote: post #64
fooker wrote: post #63

Earlier quoted context omitted.

> Even if the output is like 5-6 tok/s, that might be usable for some purposes. You'll spend ~100x more on electricity than the API cost to have it run on someone else's GPU at several hundred tokens per second. I think some sort of extreme data privacy requirement is the only situation that justifies this, but the intersection of {needs absolute data privacy, needs to run SOTA model, cannot afford GPUs} is really re…

There are a number of use cases where sending the contents of your context and prompts (and the resulting output) to a 3rd party service is off the table as an option, and people will compromise speed for data sovereignty. And not everyone's electricity is equally expensive, I pay about $0.075 USD per kWh. It would for example cost me about $48 a month of electricity (not counting cost of cooling) to run a quad socke…

Great, so the other member of the set matters for you more than cost.

Do you actually need to run the state of art model at 5 tokens per second instead of a qwen or whatever 7b or 30b model at 100 tokens per second?

Re: Kimi-K3 on HuggingFace

#76

Post by user43928 »

torginus wrote: post #17
m00dy wrote: post #4

There’s going to be a lot of competition around this model. Let’s see how low AI providers are willing to push prices.

I think the results might be underwhelming - AI providers need to turn a profit and can't subsidize, and they're working off of the commodity hardware everyone does. I wouldn't be surprised if they started offering potentiall bad quantizations with much reduced capability at lower prices (without telling the users, of course)

I would be surprised, considering that OpenRouter requires disclosing the quantization and shows automatic benchmarks to compare between providers for the same model.

Re: Kimi-K3 on HuggingFace

#77

Post by embedding-shape »

walrus01 wrote: post #69
embedding-shape wrote: post #66

Earlier quoted context omitted.

I dunno, K3 thinks a lot before it actually replies, and you might be in the ~1 tok/speed region or even "seconds / tokens", and with K3, you'd wait days if not weeks for a reply in that case. Don't get me wrong, slow is sometimes better than "not at all", but depending on the performance, it might end up way too slow to even work for batched/async jobs like that.

I agree it's very likely to be painfully slow, I very much want to see some real world results from people who try it. Early testers will inform others on whether it's even worth trying. Results very much TBD right now. I don't have a system sitting around here with 2TB of greater of RAM that isn't already committed for other uses, regretfully.

Lets say an easy response takes 32k tokens in total, and to be generous, let's say it does 1 tok/s. This is already ~9 hours, and 32k reasoning tokens isn't even that much and as mentioned, K3 probably does the longest/most reasoning/thinking out of the available open weights models today, much like GLM. Just lowering that performance to 0.5 tok/s, would lead to ~18 hours for a simple prompt to receive an answer.

And then that's just for single prompts, what about agent harnesses, where before every tool call the model could reason a bunch?

I agree with you that real world results would be interesting, but I wouldn't hold my breath nor expect it to realistically be able to be useful. Still, people should try it, for science if nothing else :)

Re: Kimi-K3 on HuggingFace

#78

Post by flaburgan »

Der_Einzige wrote: post #62
NitpickLawyer wrote: post #3

This will be interesting for a few reasons. First, depending on where the median pricing settles w/ 3rd party providers will tell us what it costs to serve a 3T model. Since it's going to be mxfp4 native, it'll take ~1.5TB of VRAM to host this, which is juuust at the limit of 8xb200s (but realistically you'll need 16x for context / throughput optimisation). Won't be cheap to host, but at least we should get some rang…

Anyone who thinks that the labs are not profitable on per token API pricing is delusional and hilariously wrong.

It all depends if you count the fixed cost of training or not. And the cost of the hardware.

Re: Kimi-K3 on HuggingFace

#79

Post by embedding-shape »

walrus01 wrote: post #73
embedding-shape wrote: post #71

Earlier quoted context omitted.

That's a lot of words to say "No, no have has seemingly done that yet with K3".

Indeed not, but I was saying there's more than ample precedent which is tested/working and actually doesn't refuse anything. Go to the Huggingface 'models' search interface and type in "heretic". Or uncensored. One example would be: https://huggingface.co/HauhauCS/Qwen3.5-122B-A10B-Uncensored...

Right, but aren't we jumping into trying to figure out solutions before someone actually checked if any solutions are needed in the first place?

Re: Kimi-K3 on HuggingFace

#80

Post by magicalhippo »

walrus01 wrote: post #50
NitpickLawyer wrote: post #3

This will be interesting for a few reasons. First, depending on where the median pricing settles w/ 3rd party providers will tell us what it costs to serve a 3T model. Since it's going to be mxfp4 native, it'll take ~1.5TB of VRAM to host this, which is juuust at the limit of 8xb200s (but realistically you'll need 16x for context / throughput optimisation). Won't be cheap to host, but at least we should get some rang…

It will be very interesting to see what kind of 'slow' performance people get from running it on a no GPU, but tons of RAM server (like a dual or quad socket xeon with 1.5 to 3TB of RAM). For the purpose of giving it longer duration tasks to generate a piece of something and come back and check on what it has done in 4 or 6 hours. Even if the output is like 5-6 tok/s, that might be usable for some purposes. Huge pric…

> running it on a no GPU, but tons of RAM server

Or from SSD using something like Colibri[1]. Not going to be quick, but at least runable.

[1]: https://github.com/JustVugg/colibri

Post reply on HN