>the prevailing view at the moment is that MLP (hence experts) in a mixture model are substantially where dense encoding of factual information resides
Yes, because that's where all the parameters are. For reference in GLM 5.2 98% of the weights are for the experts.
>The "Mixture of Parrots" work is a good jumping off point
The paper shows increased performance on knowledge dependent task while having similar reasoning capabilities. This backs up what I was saying about how the weights unlock extra performance without increasing inference costs as much as a dense model would.
>model power allocated at this point or that will recover different performance profiles
While increasing the number of weights makes the model better, where those weights are does matter in how much better the model gets and also matter in regards to the cost of training / inference. Model design is a big set of trade offs and I see MoE as a useful tool that will survive in the trade off space.
>reasoning is left on the table
Even so, if there was 2 models with an equivalent amount of reasoning ability and priced the same would you rather pay for the one with narrow knowledge or wider knowledge.
>because of the paucity of modern dense models at the near frontier,
You don't need to be at the frontier to benefit from MoE. Even open source models that are behind the frontier, benefit from being able to host experts on different machines, and scale individual, commonly used experts separately from each other. On the other end with small models you are probably resource constrained so you want to maximize the tokens generated per second. This makes going for purely dense models niche like you are saying.