Viewing profile — zozbot234

Is that an up-to-date quote? That's not a bad price for 16 TiB of RAM. Once you add the CPU cores and the storage, it's actually an amazing deal.

> The actual risk that the author does not even broach upon for investors... the thing that will actually torpedo this massive investment are the open source open weight chinese mo…

> I suspect the economics favor centralized servers, if you only look at the aggregated cost to serve X number of users' tokens. The economics of real-time, low-latency inference o…

Read disturb ought to be quite rare, especially on a fresh drive that was written only once or a handful of times (WORM-like usage). Practically, it's not likely to be an issue eve…

Yup, I hate to engage in anything that looks like a "shallow dismissal" but the project documentation seems to outright contradict itself wrt. whether it's running the model at gen…

The native weights are 4-bit for the sparse experts, and they quantize to ~80GB with limited degradation in real-world performance. That's a viable target for 64GB with SSD streami…

Reading between the lines, it's fairly clear that Mythos Preview only got its reported results in offensive cyber thanks to a highly specialized harness and humongous amounts of te…

This is not entirely true: non-constructive proofs can be fully understood within intuitionistic logic as proofs of negative statements, viz. statements about what isn't constructi…

Only the sparse experts are 4-bit in native precision, and those take up ~25GB of active params. The dense parameters' native footprint is ~115GB. So in order to infer that nativel…

Expert choice actually happens per layer, not just per token. It's not a limitation when doing inference at scale since all experts are then loaded in fast VRAM anyway. It's mostly…

> The same MTP head could also be used to speculative prefetch the expert weight residing on the SSD. If the expert weight can be preloaded before the GPU actually need them, the s…

antirez's own video (the one I referenced in my comment) shows K3 inference running on M5 Max, not M1-series silicon (which is OP). M1 series has far lower SSD read throughput and …

That's not what parent said, but this is already quite decent speed for unattended inference (overnight or even spanning multiple business days) which is arguably the right target …

> With MoE it's low probability. For even the sparsest MoE open models, having more than a handful of inferences in the batch is enough to make it more likely than not that you'll …

If you're targeting widespread local/on prem deployment which is what many open weight models are doing, that inherently limits your scale in terms of total model weights/inference…

To be clear, the restriction on "Model as a Service" past $20M yearly revenue is new to K3. K2.x had the attribution requirement for any commercial use with more than $20M monthly …

Open source AI is actually a lot less "powerful" than genuine frontier models, i.e. it has a much tighter inherent capability ceiling. This is "decelerationist" from a purely AGI-p…

If it's 4-bit native for the sparse parameters (which is the bulk of them) why would you expect it to add up to 2+TB?

The full K3 model will probably be way more than 594GB, that's more of a plausible range for Kimi 2.x. You'll probably be able to test run this model at full or near-full precision…

Chat doesn't have to literally be realtime though, that's just the model most users have settled on. You could fire off your request, let it work unattended and check back on it la…

Speculative decoding is not really worthwhile for sparsely-loaded models. You end up paying in both memory bandwith and compute (loading experts based on wrongly-predicted tokens) …

In principle, a slower inference ought to be easier to steer and course-correct. You'd always be able to look at partial results, especially with a local model that doesn't hide it…

Why does it have to be so bursty though? Just let it run multiple continuous-batched inferences overnight. This would work especially well in combination with SSD offload, and give…

For single stream inference of a MoE model, the size of active sparse parameters will matter a lot more than total parameters. This is generally around half of the reported active …

> In typical MoE model they get switched "randomly" on every token, so all experts have to be readily available. It's worse than that: a typical MoE model routes a separate set of …