Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Kimi-K3 on HuggingFace

huggingface.co

Post reply on HN
Print view View original

571–580 of 588 posts

Re: Kimi-K3 on HuggingFace

#571

Post by tripzilch »

fooker wrote: post #63
walrus01 wrote: post #50

Earlier quoted context omitted.

It will be very interesting to see what kind of 'slow' performance people get from running it on a no GPU, but tons of RAM server (like a dual or quad socket xeon with 1.5 to 3TB of RAM). For the purpose of giving it longer duration tasks to generate a piece of something and come back and check on what it has done in 4 or 6 hours. Even if the output is like 5-6 tok/s, that might be usable for some purposes. Huge pric…

> Even if the output is like 5-6 tok/s, that might be usable for some purposes. You'll spend ~100x more on electricity than the API cost to have it run on someone else's GPU at several hundred tokens per second. I think some sort of extreme data privacy requirement is the only situation that justifies this, but the intersection of {needs absolute data privacy, needs to run SOTA model, cannot afford GPUs} is really re…

> extreme data privacy requirement

so I have a a few decades worth of creative coding projects, many personal notes + writing, pictures, video clips, renders, projects, backups of the previous laptop's home folder that contains the laptop before it in the ~/dump folder and so on, spread out over several harddisks, laptops and storage media

it's a mess that I've wanted to organize for years, but I can't seem to manage it

but I might be able to sketch out a plan or task for a local multi-modal LLM to slowly chug through all the files (read-only), catalog and index them, copy to a clean external SSD, de-dupe and tidy everything up

I mean, some of this data is already highly personal, and there might be stuff I forgot was there

... is it an "extreme data privacy requirement" to not want to share running an LLM for this task, and wanna do it locally?

Re: Kimi-K3 on HuggingFace

#572

Post by vdomi »

nl wrote: post #251
Palmik wrote: post #228

Earlier quoted context omitted.

You're assuming inference providers are going to sell tokens at cost. You're also assuming that the inference providers have will optimized inference engine. I haven't seen that to be the case so far, to be honest. Take a look at GLM 5 vs GLM 5.2 pricing -- GLM 5.2 cost more despite being the same model. Take a look a look at DeepSeek, which hosts DS v4, profitably, yet others aren't able or willing to match the pric…

I think it's unclear the the DS hosted prices are profitable. AFAIK that haven't claimed that. OTOH, the multiple providers who have settled around the same price point ($3.48/M output tokens for multiple providers with good reputations) does indicate where it is profitable: https://openrouter.ai/deepseek/deepseek-v4-pro#providers

Deepseek has claimed x6 profit on inference

Re: Kimi-K3 on HuggingFace

#573

Post by amluto »

fc417fc802 wrote: post #513
amluto wrote: post #209

Earlier quoted context omitted.

> Omitting Azure, which gives some privacy for some $$$ on their models, but not at the level of high-security. If I were ranking third parties on their ability to safely handle my data without compromising it, I would rank Anthropic pretty low for things like Fable (where they more or less promise that they will misuse my data), but I want Azure pretty low in the sense that I fully expect them to be compromised. I w…

The expectation that one BigTech company has a competent security team while the other doesn't seems entirely baseless?

https://windowsforum.com/security-alerts.84/cve-2025-55241-e...

https://orca.security/resources/blog/autowarp-microsoft-azur...

https://petri.com/microsoft-azure-automation-cross-tenant-ac...

I don’t recall much in the way of AWS vulnerabilities like these, but they’re a regular occurrence on Azure.

Re: Kimi-K3 on HuggingFace

#574

Post by w4yai »

ataoz wrote: post #537
potsandpans wrote: post #536

Earlier quoted context omitted.

.

I would expect Kimi to say that "I am Kimi etc..." :D

As repeated many, many times, models usually don't know about their own name. Claude thinks he's ChatGPT, ChatGPT thinks he's DeepSeek, and Kimi thinks he's Claude. Nothing interesting to see here. Google it.

Re: Kimi-K3 on HuggingFace

#575

Post by ThunderSizzle »

mhjkl wrote: post #549
ThunderSizzle wrote: post #162

Earlier quoted context omitted.

An AMD R9700 gets 20-50 TPS at ~300 watts on 27B. 100 TPS for the 35B MOE model. And there might be some more optimizations to that as AMD software support gets better with ROCm's latest versions.

If you’re on AMD, use llamacpp with the vulkan backend. For the 35B I get around 115 tps without mtp and 150-170tps with it. Also the pcie slot you use affects it

I have issues with the vulkan backend. It seems to cause over all GUI sluggishness, but maybe that was actually Firefox. I'll give it another go.

Re: Kimi-K3 on HuggingFace

#576

Post by supjeff »

api wrote: post #375
supjeff wrote: post #369

Earlier quoted context omitted.

a "moment for humanity"? as if this shit isn't going to generate 99% slop at the cost of all we have left as a species?

Sometimes I’m not sure who is more unhinged: the total AI kool aid drinkers who think this will make us all into immortal demigods (or take over the world as it goes “foom”), or the AI doomers and haters who exaggerate everything potentially negative about it and react to it the way a 1980s Christian fundamentalist reacted to rock music. It’s a new fundamental innovation in math and CS that allows large scale lossy c…

We've had frontier models widely available for general use for 5-6 years now. Despite the promises, we haven't revolutionized anything; not society, not government, not engineering, not the environment. The only thing we've done is revolutionize the pockets of AI purveyors, suppliers and charlatans. We haven't cured disease, ended hunger, or achieved time travel. Sure, the tech is cool. But it's cool at 1b, 8b, and 30b parameters at a size that fits on consumer hardware. Hyperscale AI that's built for no purpose in particular, at any social/environmental cost, is everything that's wrong with society. That's not Christian fundamentalism or doomerism, that's recognizing the sickness at the core of the tech industry. We let the tech-billionaires delete their ESG pledges, and abandoned the fight against climate change so we can generate celebrity porn.

Re: Kimi-K3 on HuggingFace

#577

Post by dominotw »

erichocean wrote: post #335
dominotw wrote: post #323

Earlier quoted context omitted.

we need open models because they let me dehumanize roma ppl. great take by dhh. 'When gypsies appropriate public spaces, you deport them. It's not hard, it's not cruel. It's the basic logic of self-protection.'

The article did nothing of the sort: https://x.com/dhh/status/2081435971678261344 However, even if it did, it is absurd for an AI to decide what it will and won't translate.

maybe you just did cmd+f roma?

i think dhh in all his ignorance thought gypsies = roma

Re: Kimi-K3 on HuggingFace

#578

Post by aae42 »

swingboy wrote: post #102
NitpickLawyer wrote: post #3

This will be interesting for a few reasons. First, depending on where the median pricing settles w/ 3rd party providers will tell us what it costs to serve a 3T model. Since it's going to be mxfp4 native, it'll take ~1.5TB of VRAM to host this, which is juuust at the limit of 8xb200s (but realistically you'll need 16x for context / throughput optimisation). Won't be cheap to host, but at least we should get some rang…

Say a single Kimi K3 is deployed on 16 x B200s: how many concurrent users can that handle? I realize the question assumes a major simplification that everyone's prompts/sessions are the same.

In my admittedly little experience running a tiny LLM service on a single 5090 for friends, I would guess a fair bit. It depends on how many tok/sec you're targeting and how much memory you're allowing be used for context (and their context). Definitely more than 1, MAYBE an order of magnitude.

Re: Kimi-K3 on HuggingFace

#579

Post by apitman »

riknos314 wrote: post #476
apitman wrote: post #442

Earlier quoted context omitted.

Are you sure the difference is from MoE and not that 3.6 is newer?

Qwen 3.5 27B also scores higher than 3.5 122BA10B. So even in the same generation the smaller dense model outperformed the larger MOE

Ah gotcha, I hadn't noticed that. Thanks

Re: Kimi-K3 on HuggingFace

#580

Post by Kirby64 »

trollbridge wrote: post #280
cobbzilla wrote: post #187

Earlier quoted context omitted.

I’ve priced it out: max $135/month to run a dual Xeon 2U server with 3T RAM & 2x 22 core Xeon Gold. It’s the 2x 750W power supplies that ultimately determine opex. My power costs $0.124/kWh, the $135 assumes drawing maximum power continuously, and in that case, I can probably offset my heating bill a little bit in the winter, so maybe effectively a little bit lower. I don’t know if that’s 100x more than I’d pay (opex…

If you actually were running this thing at 80% or 100% load, then the first thing you'd want to is get a better PDU and then connect your servers to that (48V DC).

Why? Standard server grade PSUs are 80 Plus Platinum, exceeding 95% efficiency. The whole server is designed around 12V, especially for older gear, and replacing it with components that are 48V would be wildly expensive relative to the efficiency benefits.
Post reply on HN