Inference Optimization for MiMo v2.5: Pushing Hybrid SWA Efficiency to the Limit • Hacker News Board

Re: Inference Optimization for MiMo v2.5: Pushing Hybrid SWA Efficiency to the Limit

#41

Post by jingpostmedia » Sat, Jul 11, 2026, 1:03 PM UTC

CuriouslyC wrote: post #38

yogthos wrote: post #12

Earlier quoted context omitted.

What Chinese firms are doing makes perfect sense from the commercial perspective actually because they understand how a classic commoditization spiral works. The reality is that models themselves are general commodities and there's just not enough difference between them. A company can get ahead of others by a few months, but then the rest quickly close the gap. It's a really low margin business because there's no wa…

The Chinese labs incentives is to run inference for the world, because inference can run on the homegrown Chinese chips (giving them a guaranteed market for their hardware) and they have cheap plentiful power. The US frontier labs have an incentive to do deals with large firms to act like a contract research organization, taking royalties on creations/discoveries. Alex Karp called this out in his rant ("Why charge fo…

[flagged]

Re: Inference Optimization for MiMo v2.5: Pushing Hybrid SWA Efficiency to the Limit

#42

Post by mapontosevenths » Sat, Jul 11, 2026, 2:29 PM UTC

CuriouslyC wrote: post #36

mapontosevenths wrote: post #30

Earlier quoted context omitted.

I agree, but maybe for different reasons. I think Karpathy is right. We need models that reason, not models that memorize. Karpathy calls it a "Cognitive Core", and it's essentially a small model that learns to reason and look up the data it needs as opposed to a giant model that memorizes all the data in the world and tries to process large chunks of it all at once with every thought. I think it will be based on the…

The theorem you want to pay attention to is the no free lunch theorem. The important thing to understand there is that the larger models give you "free lunch" in the sense that you can approximate more different systems accurately at the cost of model size. If there was a Karpathy style universal solver, it wouldn't be very smart unless we scaled it up. This isn't to say that there aren't a fair amount of wasted para…

> If there was a Karpathy style universal solver, it wouldn't be very smart unless we scaled it up.

I think that the scaled up version is actually still really valuable.

Imagine being able to just add more compute as needed for any given problem until it's solved by just adding more copies of a single universal layer, without more training. Or being able to burn the individual core into silicon and just loop it as needed.

I tried to build exactly that in my personal lab once, but hit a wall made of my own incompetence and budget.

The idea was to find the parts of the manifold that did generic reasoning and then scale as needed by repeating them. It worked within individual layers (I could make the model score higher on benchmarks by repeating the reasoning extracts within individual layers), but i could never get the interfaces between layers to work again after I'd done that. I suppose it needed traing to "heal" the interface again after my brain surgery, but I didnt have the compute to manage it and moved on to the next project

I'm sure that someone who actually gets paid to do these things will figure out some version of it eventually though, because I know it can be done.

Re: Inference Optimization for MiMo v2.5: Pushing Hybrid SWA Efficiency to the Limit

#43

Post by notnullorvoid » Sat, Jul 11, 2026, 2:59 PM UTC

bee_rider wrote: post #8

LaurensBER wrote: post #2

It's really cool and interesting to see the kind of engineering that goes into Xiaomi (and Deepseeks) inference optimizations. Z.ai has also published some interesting papers although I haven't had a chance to go through them yet. It does inspire hope that the Chinese labs seem to be so open although the sceptic in me does wonder what their end game is. Surely, from a purely economic perspective it would be wiser to…

The bet could be that they’ll ultimately be able to sell hardware capable enough of running local models comfortably.

If China is able to undercut Nvidia on high performance local AI hardware, they will pop the AI bubble in a matter of days.

They wouldn't even need to make something equivalent to the latest hardware. A Chinese RTX 3090 equivalent would be enough.

Re: Inference Optimization for MiMo v2.5: Pushing Hybrid SWA Efficiency to the Limit

#44

Post by kzrdude » Sat, Jul 11, 2026, 4:09 PM UTC

TurdF3rguson wrote: post #19

alightsoul wrote: post #18

Earlier quoted context omitted.

you keep the model. it's never deprecated. with closed ai, you are forced into a new more expensive model every few months. if an open model infra provider does that, you simply switch to another one. it's not in their interest to do that.

That's not really true though, providers are deprecating models and I have at least 10 emails to prove it.

When the weights are publically available (and open to use), then there's a free market for hosting that model at least. That's not independence for me, but it's lack of vendor lock-in.

For some of the open models, there's a list of 20-30 providers of the same model on openrouter for example, as an example of the supply.

Re: Inference Optimization for MiMo v2.5: Pushing Hybrid SWA Efficiency to the Limit

#45

Post by jononor » Sat, Jul 11, 2026, 5:13 PM UTC

CuriouslyC wrote: post #38

yogthos wrote: post #12

Earlier quoted context omitted.

What Chinese firms are doing makes perfect sense from the commercial perspective actually because they understand how a classic commoditization spiral works. The reality is that models themselves are general commodities and there's just not enough difference between them. A company can get ahead of others by a few months, but then the rest quickly close the gap. It's a really low margin business because there's no wa…

The Chinese labs incentives is to run inference for the world, because inference can run on the homegrown Chinese chips (giving them a guaranteed market for their hardware) and they have cheap plentiful power. The US frontier labs have an incentive to do deals with large firms to act like a contract research organization, taking royalties on creations/discoveries. Alex Karp called this out in his rant ("Why charge fo…

Yeah China has a huge (and growing) advantage in power generation. And they have been looking to break into high-end chip manufacturing. The latter has high cost of entry and needs large scale to become viable. AI inference on own hardware would allow them to bootstrap the chip demand. Both for memory and accelerators.

Re: Inference Optimization for MiMo v2.5: Pushing Hybrid SWA Efficiency to the Limit

#46

Post by SwellJoe » Sat, Jul 11, 2026, 7:26 PM UTC

mapontosevenths wrote: post #30

SwellJoe wrote: post #16

I've been saying efficiency is the next "frontier" in AI, at least for LLMs that people use daily. Companies have started to really balk at token costs from the major providers, and there's some evidence that the cheaper Chinese models are chipping away at Anthropic/OpenAI dominance from below (as cheaper Chinese products have done in other industries for many years). I continue to think that the vendor that figures…

I agree, but maybe for different reasons. I think Karpathy is right. We need models that reason, not models that memorize. Karpathy calls it a "Cognitive Core", and it's essentially a small model that learns to reason and look up the data it needs as opposed to a giant model that memorizes all the data in the world and tries to process large chunks of it all at once with every thought. I think it will be based on the…

I don't think LLMs get us there, at least based on my understanding of them. You don't get the reasoning at a very high level without a similarly large parameter size.

Though, I don't think we're really at the maximum density of intelligence per byte. Small models of today are approaching the frontier of maybe 2-3 years ago. Gemma 4 and Qwen 3.6 in their dense versions feel quite capable, give them a search tool and they'll research a topic and often give a pretty good explanation of it. They understand more and hallucinate less than GPT-3, GPT-3.5, and maybe even GPT-4 (the first probable trillion parameter model), and their tool capabilities opened the door for agentic use.

But, those small models are still smaller thinkers. It's not a cognitive core that can do anything if given a way to look up the data, though they're more capable than a lot of folks realize.

Also, there's no reliable way to look up data, anyway. The web is poison, approaching the point where naive web search will most likely return misinformation or intentional disinformation. The source of all human knowledge must be viewed as adversarial, now, which makes having thinky machines that search the web and do stuff with the results a risky business.

Re: Inference Optimization for MiMo v2.5: Pushing Hybrid SWA Efficiency to the Limit

#47

Post by ricardobeat » Sat, Jul 11, 2026, 9:05 PM UTC

trollbridge wrote: post #26

ricardobeat wrote: post #3

Such a well written article, refreshing to read in between all the slop. I've used Mimo extensively in the past few months, can't wait to see what v3 will bring.

Are you using UltraSpeed? It's my favourite thing about MiMo.

I did, it’s amazing. Shame it’s going away and not available permanently.

Re: Inference Optimization for MiMo v2.5: Pushing Hybrid SWA Efficiency to the Limit

#48

Post by TurdF3rguson » Sat, Jul 11, 2026, 10:03 PM UTC

chronogram wrote: post #34

TurdF3rguson wrote: post #27

Earlier quoted context omitted.

I will concede a use case for hermits.

Or travel. Even in the developed world you can be without internet or slow internet. I have Gemma 4 E4B on my phone that can process audio, image, and text if I have need to.

Ok but we're talking about the concern that a provider can 3x or nerf a model. I'm pointing out that open weights providers can do that too, and your only recourse is to run it yourself at way more than 3x the price.

Re: Inference Optimization for MiMo v2.5: Pushing Hybrid SWA Efficiency to the Limit

#49

Post by zozbot234 » Sat, Jul 11, 2026, 10:20 PM UTC

ricardobeat wrote: post #3

Such a well written article, refreshing to read in between all the slop. I've used Mimo extensively in the past few months, can't wait to see what v3 will bring.

It's a very nice summary of KV cache storage requirements for the leading open weights models, but ultimately it still shows the DeepSeek V4 series winning by a huge margin on that front. And DeepSeek actually does quite well on needle-in-haystack context recall benchmark so it seems like the outstanding context compression it has isn't even impacting smarts all that much. Keep in mind that KV cache memory requirements usually has a direct impact on how much you can parallelize inference on both large-scale platforms and lower-end edge compute, so this is a very important metric.

Re: Inference Optimization for MiMo v2.5: Pushing Hybrid SWA Efficiency to the Limit

#50

Post by alightsoul » Sat, Jul 11, 2026, 11:55 PM UTC

yorwba wrote: post #39

alightsoul wrote: post #15

Earlier quoted context omitted.

its a governement mandate that states that AI research must be open source ,that's one benefit of communism

There is no such mandate. ByteDance keeps their models closed. So does iFlyTek. Qwen Max is closed as well.

But they are not groundbreaking. They are simply copies of other similar Chinese models

The mandate is worded differently from what I said