K3 releasing as open source right at the same time that people are criticizing Opus for questionable performance (there's even a thread on HN about Opus 5's problems)... this is such a flex.
Kimi-K3 on HuggingFace
271–280 of 588 posts
Re: Kimi-K3 on HuggingFace
#272Earlier quoted context omitted.
Great, so the other member of the set matters for you more than cost. Do you actually need to run the state of art model at 5 tokens per second instead of a qwen or whatever 7b or 30b model at 100 tokens per second?
>Do you actually need to run the state of art model at 5 tokens per second instead of a qwen or whatever 7b or 30b model at 100 tokens per second? Some people like doing things they want to do. Do I actually need to buy expensive pigments from europe to make paintings of flowers? My camera produces a much more accurate representation.
Very good description of it. It does seem like a bit of a rhetorical question to ask a forum that has a very high population of Linux and BSD users why they might desire to have the option to do something themselves rather than relying on an external packaged ready to go product.
Re: Kimi-K3 on HuggingFace
#273Now I hope that nvidia will host it for free :-)
if it's anything like the speeds Nvidia Nim puts Deepseek at, it'll probably be 10 tok/s or lower and timeout frequently
Re: Kimi-K3 on HuggingFace
#274Earlier quoted context omitted.
We have a lossless compression codec (working on open sourcing it over the next couple of weeks) that reduces it down to its minimum entropy -- it cannot be compressed further. On all tested large models, it's a ratio of 1.34-1.23 -- and smaller models up to 3.76x. It also increases the effective bandwidth by the same rate.
That's very interesting. Does that mean you can reduce say, a 30B class Q8 from ~30 GB down to 10 GB or less?
704gb -> 564gb; 358 gb -> 270 gb; 28.79 gb -> 7.65 gb; 439 gb -> 93 gb
It depends on the total entropy of the model. Smaller models have less entropy.
Re: Kimi-K3 on HuggingFace
#275Less than 2 hours left to the Kimi moment. It’s been more than 2 years since the DeepSeek moment that shook the world.
Re: Kimi-K3 on HuggingFace
#276[flagged]
Re: Kimi-K3 on HuggingFace
#277Earlier quoted context omitted.
There are a number of use cases where sending the contents of your context and prompts (and the resulting output) to a 3rd party service is off the table as an option, and people will compromise speed for data sovereignty. And not everyone's electricity is equally expensive, I pay about $0.075 USD per kWh. It would for example cost me about $48 a month of electricity (not counting cost of cooling) to run a quad socke…
As someone who has worked in two industries that are at the maximal end of data sensitivity and privacy this comes across as a tinfoil hat issue not a real business requirement. In such cases we've always found ways to trade dollars for the privacy we need without having to run our own inference at excruciating slow speeds.
Interesting. So nobody would have had a problem with you running stuff on Chinese AI providers?
I have some inference I simply don't want to run on OAI, Anthropic, or Google because I don't want to run afoul of their "rules" and end up with a banned account, and this situation is only getting worse when it comes to doing fairly basic tasks like trying to secure your app against security problems.
Re: Kimi-K3 on HuggingFace
#278Is there any (near future) technology that would permit burning this terrabyte into some kind of ROM chip?
Given how fast models are improving, burning the weights into actual ROM is prohibitively expensive if you need (or want) to replace the chip every couple of months.
The alternative is on-TPU flash for storing the weights.
Re: Kimi-K3 on HuggingFace
#279Wait, so I can download it and run it locally now?? Wow... But it probably won't work on my computer, right?
Yes it will. Buy a 4TB nvme, allocate 3TB as swap, and run a gpu emulator on your cpu.
Re: Kimi-K3 on HuggingFace
#280Earlier quoted context omitted.
> Even if the output is like 5-6 tok/s, that might be usable for some purposes. You'll spend ~100x more on electricity than the API cost to have it run on someone else's GPU at several hundred tokens per second. I think some sort of extreme data privacy requirement is the only situation that justifies this, but the intersection of {needs absolute data privacy, needs to run SOTA model, cannot afford GPUs} is really re…
I’ve priced it out: max $135/month to run a dual Xeon 2U server with 3T RAM & 2x 22 core Xeon Gold. It’s the 2x 750W power supplies that ultimately determine opex. My power costs $0.124/kWh, the $135 assumes drawing maximum power continuously, and in that case, I can probably offset my heating bill a little bit in the winter, so maybe effectively a little bit lower. I don’t know if that’s 100x more than I’d pay (opex…
If you actually were running this thing at 80% or 100% load, then the first thing you'd want to is get a better PDU and then connect your servers to that (48V DC).