Why is everyone's pricing the same? licensing agreements with moonshot as an inference provider?
Kimi-K3 Technical Report [pdf]
151–160 of 199 posts
Re: Kimi-K3 Technical Report [pdf]
#152Earlier quoted context omitted.
And hire 2 or 3 dev ops to keep it running ? That another 400 to 700k. It becomes your problem and not someone else’s. However, I don’t trust hosted LLMs for anything that needs to be private.
> I don’t trust hosted LLMs for anything that needs to be private I'd update this to 'I don’t LLMs for anything that needs to be private' What's to prevent the LLM from sliding a heavily obfuscated binary blob into the application that does nefarious things? If you aren't creating the LLM itself from scratch, I don't feel it can be trusted.
> What's to prevent the LLM from
A NN per se is a file... The executable that runs it can "act"...
Re: Kimi-K3 Technical Report [pdf]
#153Earlier quoted context omitted.
And hire 2 or 3 dev ops to keep it running ? That another 400 to 700k. It becomes your problem and not someone else’s. However, I don’t trust hosted LLMs for anything that needs to be private.
> I don’t trust hosted LLMs for anything that needs to be private I'd update this to 'I don’t LLMs for anything that needs to be private' What's to prevent the LLM from sliding a heavily obfuscated binary blob into the application that does nefarious things? If you aren't creating the LLM itself from scratch, I don't feel it can be trusted.
Why do you feel that creating the LLM from scratch is sufficient to trust it? Are you suggesting that you personally would read all 15 trillion tokens (plus every single agentic trade used in RL, along with its relative advantage in the batch) and personally guarantee that gradient descent would train a model which would not exfiltrate your corporate data?
Or that perhaps you have a perfect alignment algorithm which you are unwilling to share with the broader research community (evil)?
Re: Kimi-K3 Technical Report [pdf]
#154Re: Kimi-K3 Technical Report [pdf]
#155Any guess on the pre-training tokens/flops they consumed?
Technical report has a graph vs Kimi K2 with 1e21 FLOPs (but they don't claim that's the entirety of pretraining)
1e21 flops is hilariously wrong. for reference the llama 3 8B model (https://arxiv.org/pdf/2407.21783) used 10 times that many flops. This model is 350x bigger in total params and 12x bigger in active params and was trained on 3x the data.
Re: Kimi-K3 Technical Report [pdf]
#156Earlier quoted context omitted.
Understood but sharing your existing devops resources with this will soon become a bottleneck especially when any major downtime will keep several engineers (and long-running agents) blocked from any meaningful work until availability improves.
Not my experience, from an SMB that maintains its own hardware and services. You have a certain contingent of competent engineers who distribute their work across projects, and it generally works out fine. Or course you plan with some redundancy and fall-back plans in your systems.
The top poster mentioned LLM spend of millions/month to justify the estimated capex of $6m to self-host Kimi on own infra.
Add to this number another $1.5m/yr in opex, so not sure I’d call such an enterprise wealthy enough to spend those kinds of sums on LLMs an “SMB”.
Re: Kimi-K3 Technical Report [pdf]
#157Earlier quoted context omitted.
How many companies host and serve models via API and have a $20M+ revenue? Going to be pretty straightforward to catch offenders.
Demonstrating a company is using this model in court seems non-trivial, no? Or am I interpreting this post incorrectly.
Not at all. It is easy to bake in a specific response to a given prompt in the model.
Re: Kimi-K3 Technical Report [pdf]
#158Earlier quoted context omitted.
From the paper (page 6 with a comparison to GLU and SwiGLU) they are not using tanh directly (i.e. f(x) = tanh(x)) but: f_gate(b,x) = b * tanh(x / b) * sigmoid(x) f_up(b,x) = b * tanh(x / b) Looking at the graph I wonder if this is to try and get the best of both GLU (better representation at higher values of x >~ 5) and SwiGLU (the value bump just before 0).
Which part of that is doing gating? I thought gate generally looks like f_gate(x, y) = f(x) * y for some f. In your case b is a constant hyperparameter though, e.g. f_gate(x) = 4 * tanh(x / 4) * sigmoid(x) So where's the gate?
In the linked "Kimi-K3 Technical Report [pdf]" paper, section 2.3 (Stable LatentMoE, p6) has the table with those equations on (top of p7, using β_1 for the gate branch and β_2 for the up branch). They talk specifically about the function in section 2.3.2 (Sigmoid Tanh Unit GLU, bottom of p7).
Figure 2 (p3) has the architecture diagram for K3 (which is the same one on the Kimi K3 blog post https://www.kimi.com/blog/kimi-k3). AFAICT, that diagram along with section 2.3 should answer your question.
Note: I'm not familiar with LLM/NN architecture to answer it more precisely than that; I only have a surface level understanding from watching various YouTube channels like 3B1B and Welch Labs on the subject.
Re: Kimi-K3 Technical Report [pdf]
#159Back of the envelope calculation (could be off, correct me if I am) If you are a large enough company that spends million+ on inference a month, it makes sense to buy a GB300 rack ($6M on top range from what I could find) which has 20.7 TB. Since the model is mixed trained (MXFP4), you would need less than 10% of the rack's memory to serve the full model. Aggregate HBM bandwidth: 576 TB/s. You can run over 6000 paral…
We're running Kimi 2.8 on a $107k server and getting around 50k tokens per second or better on most things. We've already saved money compared to last years token cost on Claude/Gemini
Are you developing software? Is most of it used on a coding agent? (Like Claude Code or ChatGPT Codex?) If so, what coding agent do you use? If you're not developing software what do you use it for (roughly)?
Re: Kimi-K3 Technical Report [pdf]
#160Earlier quoted context omitted.
I wonder how they'll figure out who to target for litigation when this license is violated.
How many companies host and serve models via API and have a $20M+ revenue? Going to be pretty straightforward to catch offenders.
Yep. Cursor’s Composer 2 model is a good example, though it is not clear if they entered into an agreement with Moonshot before they got found out in March this year [1] or after.