Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Kimi-K3 Technical Report [pdf]

github.com

Post reply on HN
Print view View original

161–170 of 199 posts

Re: Kimi-K3 Technical Report [pdf]

#161

Post by freediddy »

FuriouslyAdrift wrote: post #135
GodelNumbering wrote: post #60

Back of the envelope calculation (could be off, correct me if I am) If you are a large enough company that spends million+ on inference a month, it makes sense to buy a GB300 rack ($6M on top range from what I could find) which has 20.7 TB. Since the model is mixed trained (MXFP4), you would need less than 10% of the rack's memory to serve the full model. Aggregate HBM bandwidth: 576 TB/s. You can run over 6000 paral…

We're running Kimi 2.8 on a $107k server and getting around 50k tokens per second or better on most things. We've already saved money compared to last years token cost on Claude/Gemini

how many requests per second can the server take?

Re: Kimi-K3 Technical Report [pdf]

#163

Post by ajju »

FuriouslyAdrift wrote: post #135
GodelNumbering wrote: post #60

Back of the envelope calculation (could be off, correct me if I am) If you are a large enough company that spends million+ on inference a month, it makes sense to buy a GB300 rack ($6M on top range from what I could find) which has 20.7 TB. Since the model is mixed trained (MXFP4), you would need less than 10% of the rack's memory to serve the full model. Aggregate HBM bandwidth: 576 TB/s. You can run over 6000 paral…

We're running Kimi 2.8 on a $107k server and getting around 50k tokens per second or better on most things. We've already saved money compared to last years token cost on Claude/Gemini

What is your config?

Re: Kimi-K3 Technical Report [pdf]

#164

Post by JimmaDaRustla »

999900000999 wrote: post #70
GodelNumbering wrote: post #60

Back of the envelope calculation (could be off, correct me if I am) If you are a large enough company that spends million+ on inference a month, it makes sense to buy a GB300 rack ($6M on top range from what I could find) which has 20.7 TB. Since the model is mixed trained (MXFP4), you would need less than 10% of the rack's memory to serve the full model. Aggregate HBM bandwidth: 576 TB/s. You can run over 6000 paral…

And hire 2 or 3 dev ops to keep it running ? That another 400 to 700k. It becomes your problem and not someone else’s. However, I don’t trust hosted LLMs for anything that needs to be private.

Why would a singe system require 3 full-time dev ops?

Re: Kimi-K3 Technical Report [pdf]

#165

Post by niltecedu »

999900000999 wrote: post #124
margalabargala wrote: post #122

Earlier quoted context omitted.

Yeah but you don't need two such people, or even one, dedicated to this single rack. A company of the size that this is worthwhile for, probably has dedicated devops on staff already and can add this rack to the inventory with no additional staff.

Who is going to upgrade the models ? Who is going to fix it when the api does something weird ? Who is going to proactively make sure it’s not overheating? Chat GPT has enterprise contracts for a reason.

My company is not even the same ballpark, but even we already have people for it. And doesn't include the fact that you can get a colo location and just a MSP or a contractor to do it for you

Re: Kimi-K3 Technical Report [pdf]

#166

Post by wongarsu »

mdp2021 wrote: post #150
FuriouslyAdrift wrote: post #135

Earlier quoted context omitted.

We're running Kimi 2.8 on a $107k server and getting around 50k tokens per second or better on most things. We've already saved money compared to last years token cost on Claude/Gemini

> We're running Kimi 2.8 on a $107k server Equipped with what? Is it CPU based inference, a mix...?

Not GP, but my educated guess is that they are running a system with between 4 and 6 MI325 or MI355x or similar AMD GPUs. With the 50k tps as the total figure for all parallel requests. Those cards have a lot of memory for their price, allowing you to push to really high batch sizes while still having a large context size for each request

Re: Kimi-K3 Technical Report [pdf]

#167

Post by [deleted] »

bityard wrote: post #58
throwaway27448 wrote: post #15

Earlier quoted context omitted.

I wonder how they'll figure out who to target for litigation when this license is violated.

Software licenses aren't enforced through litigation as much as they are enforced through the _threat_ of litigation and legal risk. In other words, pretty much every company pays lawyers to minimize legal risk. Those lawyers inevitably look at all the contracts, agreements, and software licenses, and tell the C-suite what to do in order to keep their legal exposure as low as possible. "Don't violate other companies'…

[deleted]

Re: Kimi-K3 Technical Report [pdf]

#168

Post by DennisP »

bityard wrote: post #58
throwaway27448 wrote: post #15

Earlier quoted context omitted.

I wonder how they'll figure out who to target for litigation when this license is violated.

Software licenses aren't enforced through litigation as much as they are enforced through the _threat_ of litigation and legal risk. In other words, pretty much every company pays lawyers to minimize legal risk. Those lawyers inevitably look at all the contracts, agreements, and software licenses, and tell the C-suite what to do in order to keep their legal exposure as low as possible. "Don't violate other companies'…

Normal software is copyrighted. Model weights don't seem to be copyrighted, as another comment pointed out here: https://news.ycombinator.com/item?id=49073127

If weights aren't copyrighted then there's no IP to license, other than the bit of software that runs the whole thing and it's probably not too hard to write that yourself. There might even be open source reimplementations soon.

Lots of companies are willing to push the boundaries of IP when there's a gray area. One example is Google's implementation of a Java API when they made Android, for which they got sued, and won their case. Another example is the AI companies training their models on copyrighted material.

Re: Kimi-K3 Technical Report [pdf]

#169

Post by jujube3 »

whimsicalism wrote: post #111
embedding-shape wrote: post #98

Earlier quoted context omitted.

Don't make me add more of my thoughts to the comment, I can still edit it. Deeply fun "care about improving people's lives" quote on your user page :p

how is 2025 two years ago? GPT-120b is not even close to the biggest recent American release, etc. etc.

2025 is 10 years ago in AI time.

Re: Kimi-K3 Technical Report [pdf]

#170

Post by DiskoHexyl »

margalabargala wrote: post #142
999900000999 wrote: post #124

Earlier quoted context omitted.

Who is going to upgrade the models ? Who is going to fix it when the api does something weird ? Who is going to proactively make sure it’s not overheating? Chat GPT has enterprise contracts for a reason.

The same person managing the company's email accounts and whatnot. I didn't say it's fire-and-forget. I'm saying all that is maybe a day of work every 3 months.

It doesn’t really work like that.

The companies which have the will and the budget to host their own LLMs typically require a ton of other, much smaller, models as well. They have internal security requirements, guardrails, audit, critical workflows start depending on your onprem setup, downtime is now something that’s not even allowed. There’s going to be a zoo of tooling, lots of bespoke work with internal clients who have no clue about docker, but now want their vibe-coded app to access a model they downloaded yesterday, and this model better be served and monitored 24/7 because now C-levels use it.

No one is going to budget several millions to then look at an email admin who hears ‘cuda’ for the first time in their life and ask them to just support the entire thing somehow.

And god forbid it’s an AMD setup.

This obviously isn’t relevant for a 10-person startup and their second-hand xeon with a single H100.

From my experience only the companies which are REALLY interested in privacy and data security bother with hosting their own models

Post reply on HN