Kimi-K3 on HuggingFace

Post by holoduke » Mon, Jul 27, 2026, 8:39 PM UTC

walrus01 wrote: post #50

NitpickLawyer wrote: post #3

This will be interesting for a few reasons. First, depending on where the median pricing settles w/ 3rd party providers will tell us what it costs to serve a 3T model. Since it's going to be mxfp4 native, it'll take ~1.5TB of VRAM to host this, which is juuust at the limit of 8xb200s (but realistically you'll need 16x for context / throughput optimisation). Won't be cheap to host, but at least we should get some rang…

It will be very interesting to see what kind of 'slow' performance people get from running it on a no GPU, but tons of RAM server (like a dual or quad socket xeon with 1.5 to 3TB of RAM). For the purpose of giving it longer duration tasks to generate a piece of something and come back and check on what it has done in 4 or 6 hours. Even if the output is like 5-6 tok/s, that might be usable for some purposes. Huge pric…

Post by fc417fc802 » Mon, Jul 27, 2026, 8:46 PM UTC

amluto wrote: post #209

frognumber wrote: post #190

Earlier quoted context omitted.

Having worked in / adjacent several such industries, a lot of the question depends on scale. A trillion-dollar business can easily trade dollars for the privacy. A business with $1M to spend won't even get a phone call with OpenAI or Anthropic, who were the only* previous players in town for doing this. Worst-case example: Bootstrapped startup working in military. It's also the case that an open model enables many mo…

> Omitting Azure, which gives some privacy for some $$$ on their models, but not at the level of high-security. If I were ranking third parties on their ability to safely handle my data without compromising it, I would rank Anthropic pretty low for things like Fable (where they more or less promise that they will misuse my data), but I want Azure pretty low in the sense that I fully expect them to be compromised. I w…

Post by ComputerGuru » Mon, Jul 27, 2026, 8:56 PM UTC

walrus01 wrote: post #64

fooker wrote: post #63

Earlier quoted context omitted.

> Even if the output is like 5-6 tok/s, that might be usable for some purposes. You'll spend ~100x more on electricity than the API cost to have it run on someone else's GPU at several hundred tokens per second. I think some sort of extreme data privacy requirement is the only situation that justifies this, but the intersection of {needs absolute data privacy, needs to run SOTA model, cannot afford GPUs} is really re…

There are a number of use cases where sending the contents of your context and prompts (and the resulting output) to a 3rd party service is off the table as an option, and people will compromise speed for data sovereignty. And not everyone's electricity is equally expensive, I pay about $0.075 USD per kWh. It would for example cost me about $48 a month of electricity (not counting cost of cooling) to run a quad socke…

Post by seafoam » Mon, Jul 27, 2026, 9:10 PM UTC

mambru wrote: post #508

flutas wrote: post #505

Earlier quoted context omitted.

I'm in Arkansas and get rates fairly similar as quoted. From my last bill > KWH USAGE 2590 - $183.37 There's a base customer cost of $18 on top of that, but yeah ~$0.077/kWh taxes included.

Is that for a month or a year?

Post by ClumsyPilot » Mon, Jul 27, 2026, 9:22 PM UTC

fc417fc802 wrote: post #513

amluto wrote: post #209

Earlier quoted context omitted.

> Omitting Azure, which gives some privacy for some $$$ on their models, but not at the level of high-security. If I were ranking third parties on their ability to safely handle my data without compromising it, I would rank Anthropic pretty low for things like Fable (where they more or less promise that they will misuse my data), but I want Azure pretty low in the sense that I fully expect them to be compromised. I w…

The expectation that one BigTech company has a competent security team while the other doesn't seems entirely baseless?