I love LLMs, I hate hype

Remember when George was an iconoclast hacker and not the lapdog of AI oligarchs?

This will also create a regulatory issue

The US government thinks they can dictate who can access "Mythos-level" (whatever that is) LLMs. But what will happen when this can be run on consumer hardware?

I guess this will be yet another vector too attack open computing and the idea that people can a) own computers, and b) choose what software they run on their own computers

A slight defense of any qualifiers, mine or others. I am a sole developer, who formed a "box" as you put it, to house my assets. I'm the sole director of it. That doesn't mean my box is at war with users. To be clear, a merchant is a middleman and some merchants have taken the goods produced by the builder and built a middleman moat. Cloudflare, Anthropic, and the list goes on. Geohot doesn't do this. He builds and sells what he builds. That makes him more a craftsman, not a merchant. Merchants market other's work and profit. Craftsman, or builders, build and grow their customer base organically.

I think we're fundamentally reading from the same sheet of music but drawing different conclusions.

Mixture models have compute advantages at training time, everyone agrees about that, that was the original rationale (popularized at the time with `mixtral-8x7B` among others). This seems to be likely to remain an economically relevant strategy for (especially) pretrain: in a training setting you have already paid for fast interconnect at scale, a dense architecture doesn't buy you anything in a big pretrain and it costs you a lot of traffic and to a lesser degree batch size under the roofline. The heavy, FLOPs intense, interconnect intense parts of training run benefit enormously from MoE: I don't dispute that though I suspect we are well into convergence on the target precision (4) and the target format (NVFP4 or similar). At some point the whole Internet is in the pretrain at the terminal generalizing precision, the pretrains of the various labs start to look a lot alike, and the sauce remains in the later parts of training along with the proprietary data sets and what not. Frankly all the labs would benefit from standardizing the Common Crawl recoverable pretrain to greater or lesser degree, it would lower everyone's costs without changing the competitive landscape much. But that's my prediction/opinion, that's why I said "suspect".

The evidence is suggestive if not fully conclusive that mixture models are strictly losing in most regimes during inference: the exemplar of Qwen3.6-27B (which outperforms Alibab's own mixture model at ~ ten times the size) is very suggestive, and it's not the only argument for this. Because most/all modern MoE requires the activations of the previous layer before routing the subsequent experts, you are pretty much paying for the HBMe3 or GDDR7 to hold the whole thing even though some small fraction of it is under your roofline on any given token or draft verification.

This is grossly wasteful under all trajectories (even parity at N parameters between dense and MoE, which we have evidence is off by 1-2 orders of magnitude): in a "local LLM" setting (from bedroom to regional office, anything other than an NVL72 or Ironwood rack) you are sharply constrained by both total available accelerator DRAM and accelerator memory bandwidth: you are probably not getting under your roofline even with pretty slow tensor units (e.g. GB10). This use case matters and looks like it's going to matter more and more over time (the GB10 in particular is going into about a gigaton of RTX Spark laptops next year). In a datacenter setting, you're paying for extreme interconnect (training class hardware setups basically) for pure forward pass that wouldn't otherwise need training optimized gear. Even multi-trillion parameter dense models can fit in 8x or 16x RTX 6000 Pro style setups (hell, there's a DGX branded one) and with all of the geometry, tiling, scheduling, and interconnect needs mapped out up front, the design space is really forgiving on all manner of tensor parallelism, pipelining, KV cache sharding, it's a very friendly constraint space up to like, 3-5T parameters. MoE at inference time works for two groups of people: people who are willing to page experts in per token/draft batch in local LLM settings (not probably ever going to be mainstream, people really dislike that level of slow), and the vendors of extreme performance interconnect i.e. vendors selling training-class equipment as necessary for inference. And you pay in so many other ways: grouped GEMM is no one's idea of a good time, the kernels are fiendishly difficult, therefore they are not abundant, it's no fun.

The path forward here is non-obvious, and I don't claim to have it all figured out. But since you seem interested enough to carry the conversation past the pleasantries, a more substantial version of my thoughts on the matter can be found at: https://cdn.s4.gl/preprints/pairwise-interlock.pdf

This is clearest when we create open source communities that are not centered around a code project.

Stack overflow was an amazing open documentation community. Even if the code snippets were less amazing.

One central fault of GNU is their focus on their code and their problematic Jiu-Jitsu attempt to force reluctant companies into doing open-source.

Then again the history of Chromium is less about community - I'm not sure where that fits in my world modeling.

(reedited - clarified words)