Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Kimi-K3 Technical Report [pdf]

github.com

Post reply on HN
Print view View original

141–150 of 199 posts

Re: Kimi-K3 Technical Report [pdf]

#141

Post by yorwba »

reissbaker wrote: post #83
Smaug123 wrote: post #64

Earlier quoted context omitted.

If your worldview is “most of the progress is made by closed labs, then open labs fast-follow” (which isn’t implausible given the documented distillation of Fable), and further that open labs cannot make make meaningful progress vs the closed labs except by fast-following and that they won’t pick up the ability to make progress after the closed labs are gone, then driving closed labs out of business slows down overal…

I think it's pretty hard to hold that worldview: Anthropic couldn't ship a reasoning model until they copied DeepSeek R1's homework, and they've all copied DS-style super-sparse MoEs at this point too.

With slightly different cherry-picking, you could equally well claim that DeepSeek couldn't ship a reasoning model until they copied the idea from OpenAI's o1-preview, and they also copied MoEs from Google Brain/Jagellonian University https://arxiv.org/abs/1701.06538 way back in 2017, too!

But ultimately these were ideas floating around in the air, if one group hadn't done the experiment, someone else would have.

Re: Kimi-K3 Technical Report [pdf]

#142

Post by margalabargala »

999900000999 wrote: post #124
margalabargala wrote: post #122

Earlier quoted context omitted.

Yeah but you don't need two such people, or even one, dedicated to this single rack. A company of the size that this is worthwhile for, probably has dedicated devops on staff already and can add this rack to the inventory with no additional staff.

Who is going to upgrade the models ? Who is going to fix it when the api does something weird ? Who is going to proactively make sure it’s not overheating? Chat GPT has enterprise contracts for a reason.

The same person managing the company's email accounts and whatnot.

I didn't say it's fire-and-forget. I'm saying all that is maybe a day of work every 3 months.

Re: Kimi-K3 Technical Report [pdf]

#144

Post by shrubble »

999900000999 wrote: post #70
GodelNumbering wrote: post #60

Back of the envelope calculation (could be off, correct me if I am) If you are a large enough company that spends million+ on inference a month, it makes sense to buy a GB300 rack ($6M on top range from what I could find) which has 20.7 TB. Since the model is mixed trained (MXFP4), you would need less than 10% of the rack's memory to serve the full model. Aggregate HBM bandwidth: 576 TB/s. You can run over 6000 paral…

And hire 2 or 3 dev ops to keep it running ? That another 400 to 700k. It becomes your problem and not someone else’s. However, I don’t trust hosted LLMs for anything that needs to be private.

As mentioned, it is a "large enough company" already; they have full time sysadmins running things.

Adding another rack beside the VMWare cluster, managing any storage/networking issues, etc. will be incremental costs; they already have a pager (probably not a pager not anymore just an app on their phone) like rotation schedule etc.

Re: Kimi-K3 Technical Report [pdf]

#145

Post by parodysbird »

embedding-shape wrote: post #57
parodysbird wrote: post #52

Earlier quoted context omitted.

GPL licenses also restrict what you can do with it...

Beyond reciprocity which is the entire point of that license, what restrictions does it come with? I guess you could say that it has a restriction of adding new restrictions, but surely that's not what you're talking about?

Copyleft licenses have restrictions. Now, those restrictions are definitely in promotion of the normative ethics that the FOSS community holds, but they are nonetheless restrictions.

Re: Kimi-K3 Technical Report [pdf]

#147

Post by htrp »

sosodev wrote: post #99
eamag wrote: post #20

Can someone explain what are teachers in Multi-Teacher On-Policy Distillation? I can imagine math, coding and other verifiable domains, but they also have biology? Is it where distillation from bigger models come in?

They reference https://thinkingmachines.ai/blog/on-policy-distillation/ If I understand correctly, it's distillation via having a teacher model score each of the student's tokens for a problem based on their own probabilities of generating that token at each step in the sequence. The reward/loss is then applied as RL. The multi-teacher bit seems to imply they're distilling from multiple models. It's light on the deta…

> which OpenAI seems to allow via API but not Anthropic

Does openai still allow logprobs in their current gen models?

Re: Kimi-K3 Technical Report [pdf]

#149

Post by w0m »

999900000999 wrote: post #70
GodelNumbering wrote: post #60

Back of the envelope calculation (could be off, correct me if I am) If you are a large enough company that spends million+ on inference a month, it makes sense to buy a GB300 rack ($6M on top range from what I could find) which has 20.7 TB. Since the model is mixed trained (MXFP4), you would need less than 10% of the rack's memory to serve the full model. Aggregate HBM bandwidth: 576 TB/s. You can run over 6000 paral…

And hire 2 or 3 dev ops to keep it running ? That another 400 to 700k. It becomes your problem and not someone else’s. However, I don’t trust hosted LLMs for anything that needs to be private.

> I don’t trust hosted LLMs for anything that needs to be private

I'd update this to

'I don’t LLMs for anything that needs to be private'

What's to prevent the LLM from sliding a heavily obfuscated binary blob into the application that does nefarious things? If you aren't creating the LLM itself from scratch, I don't feel it can be trusted.

Re: Kimi-K3 Technical Report [pdf]

#150

Post by mdp2021 »

FuriouslyAdrift wrote: post #135
GodelNumbering wrote: post #60

Back of the envelope calculation (could be off, correct me if I am) If you are a large enough company that spends million+ on inference a month, it makes sense to buy a GB300 rack ($6M on top range from what I could find) which has 20.7 TB. Since the model is mixed trained (MXFP4), you would need less than 10% of the rack's memory to serve the full model. Aggregate HBM bandwidth: 576 TB/s. You can run over 6000 paral…

We're running Kimi 2.8 on a $107k server and getting around 50k tokens per second or better on most things. We've already saved money compared to last years token cost on Claude/Gemini

> We're running Kimi 2.8 on a $107k server

Equipped with what? Is it CPU based inference, a mix...?

Post reply on HN