Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Qwen3.8-Flash-Next

qwen.ai

Post reply on HN
Print view View original

201–210 of 246 posts

Re: Qwen3.8-Flash-Next

#201

Post by a11r »

schopra909 wrote: post #107

Can someone explain the intuition behind the en-gram idea? I know DeepSeek published a paper about it a few months ago and the Gemma models have a lightweight version of it; but it hasn’t clicked for me yet

Quoting RGFusion from Reddit: LLMs run into an issue where the further you train a model, the more it overwrites facts with generalized concepts. You need the model to be able to do both. Intelligence arises from generalization, but without accurate information the model will hallucinate.

The engram table allows for a low-computational method of fact-recall. You can think of it like a better form of RAG, where the data doesn't take up any of your context window and it's injected deeper into the model's layers, freeing the lower layers to carry out abstraction. This results in better "focus" for the model, both in regards to its intelligence and context recall.

Basically, they've separated the specificity-critical portions of the models memory into a parameter space that doesn't need fast compute (you can run it on system RAM) and allows the model to be trained on higher volumes of data without ruining its knowledge-base.

https://www.reddit.com/r/LocalLLaMA/comments/1vy6smx/comment...

Re: Qwen3.8-Flash-Next

#202

Post by mediaman »

aaa_aaa wrote: post #157
mediaman wrote: post #146

Earlier quoted context omitted.

You can make the other argument that China subsidizes the price and that they can't be profitable at this pricing level. From an industrial strategy standpoint, they already do this for many other industries with huge subsidized state loans. So we can go round and round on this, each with our made-up objections about how it's temporary or unrealistic or impossible or whatever, or we can just accept the prices as list…

Private companies cannot play that game too long. Profit from current state of AI is a mirage and sooner or later stuff will hit the fan.

Just look at the prices that inference providers charge for small models. The argument that these unit economics are negative is trivial to disprove.

DeepInfra sells DS v4-flash at 0.08 in, $0.18 out. Gemma4 they sell for $0.07 in, $0.34 out. OpenAI's price for luna is $0.20 in, $1.20 out.

Why would you assume OpenAI is somehow uniquely incompetent at making small, fast models? And that they're worse at serving it than DeepInfra? Any observer can see they are making money here.

I never understand why people who are convinced there is a big con just don't check market prices and see if there's money to be made.

That doesn't mean their business is great -- they're losing tons of money, but it's because they spend too much on fixed costs, and they can't stop spending money on training next generation models with no end in sight, not because the inference is margin negative, which is a flimsy idea that just clouds the actual business issue.

Re: Qwen3.8-Flash-Next

#203

Post by hedgehog »

cmrdporcupine wrote: post #189
hedgehog wrote: post #188

Earlier quoted context omitted.

Ok, at 50k context its about 126 prefill, 13 generation.

Oh, interesting. So beats on prefill and matches at decode. MTP will help you a bit once you have it. I've got a lot of work to do to optimize prefill. Kinda wish I had a Strix Halo here to play with as well.

I just took a minute to look at your eider repo, very cool. It looks like most of the code outside the kernels and immediately surrounding plumbing would work well on AMD APUs, and probably also on Apple and newer Intel.

Re: Qwen3.8-Flash-Next

#204

Post by mapontosevenths »

mdp2021 wrote: post #194
hedgehog wrote: post #162

Earlier quoted context omitted.

Separating knowledge from reasoning so you only pay for what you use is a big rationale for MoE, the big problem being MoE training has historically been hard to get right. In a dense model every single token you're paying a cost to determine whether you're now talking about the flavor of durian.

Sure, the "6b subset" can be more knowledgeable on its area than a whole 27b generalist (and more efficient), but where is the simulated Intelligence encoded? A 6b subset as or more intelligent than a 27b raises the question of how metacognition skills are stored.

MOEs are built by training a second "router" model to identify which parts matter inside the dense model.

Think of MOE as ignoring noise, rather than a more efficient encoding of the data we teach it which ones can be ignored. Turning down the noise actually sharpens the results sometimes.

Modern LLM's are wildly inefficient.

Re: Qwen3.8-Flash-Next

#205

Post by cmrdporcupine »

hedgehog wrote: post #203
cmrdporcupine wrote: post #189

Earlier quoted context omitted.

Oh, interesting. So beats on prefill and matches at decode. MTP will help you a bit once you have it. I've got a lot of work to do to optimize prefill. Kinda wish I had a Strix Halo here to play with as well.

I just took a minute to look at your eider repo, very cool. It looks like most of the code outside the kernels and immediately surrounding plumbing would work well on AMD APUs, and probably also on Apple and newer Intel.

Thanks; I have another, currently private, repo that targets both pure CPU inference and wgpu (to vulkan.) It's somewhat similar but... different. Shares some common pieces but needs to be refactored to share more.

But it's been hard to make it competitive with CUDA. At least on this Spark and my only non-NVIDIA machine (which only has 16GB unified relatively slow RAM.)

Re: Qwen3.8-Flash-Next

#206

Post by twobitshifter »

schopra909 wrote: post #107

Can someone explain the intuition behind the en-gram idea? I know DeepSeek published a paper about it a few months ago and the Gemma models have a lightweight version of it; but it hasn’t clicked for me yet

Ngram is compressing several layers of multiplication to a lookup which negates the need to have the same model depth and reduces the model size that must be loaded.

Re: Qwen3.8-Flash-Next

#207

Post by hedgehog »

cmrdporcupine wrote: post #205
hedgehog wrote: post #203

Earlier quoted context omitted.

I just took a minute to look at your eider repo, very cool. It looks like most of the code outside the kernels and immediately surrounding plumbing would work well on AMD APUs, and probably also on Apple and newer Intel.

Thanks; I have another, currently private, repo that targets both pure CPU inference and wgpu (to vulkan.) It's somewhat similar but... different. Shares some common pieces but needs to be refactored to share more. But it's been hard to make it competitive with CUDA. At least on this Spark and my only non-NVIDIA machine (which only has 16GB unified relatively slow RAM.)

I know a little bit about this problem space from previous work (we were working on performance-portable deep learning back around 2016). The infrastructure has improved but as far as I can tell not many teams have really "squeezed the toothpaste tube" and worked through performance issues systematically. These days a small team and robots can probably do it though.

Re: Qwen3.8-Flash-Next

#208

Post by cmrdporcupine »

hedgehog wrote: post #207
cmrdporcupine wrote: post #205

Earlier quoted context omitted.

Thanks; I have another, currently private, repo that targets both pure CPU inference and wgpu (to vulkan.) It's somewhat similar but... different. Shares some common pieces but needs to be refactored to share more. But it's been hard to make it competitive with CUDA. At least on this Spark and my only non-NVIDIA machine (which only has 16GB unified relatively slow RAM.)

I know a little bit about this problem space from previous work (we were working on performance-portable deep learning back around 2016). The infrastructure has improved but as far as I can tell not many teams have really "squeezed the toothpaste tube" and worked through performance issues systematically. These days a small team and robots can probably do it though.

At my day job I may get access to big AMD AI iron in a couple months (to do research/performance tuning with). That could be interesting. Though that's likely to be of a very different shape from consumer Vulkan. I'd still like to have a Strix Halo to futz with. But I'll wait for RAM prices to drop. (Hah!). I do have an older BC250 board lying around but that only has 16GB RAM.

I got prefill up to 190 tok/sec just now, BTW.

Re: Qwen3.8-Flash-Next

#209

Post by itishappy »

user43928 wrote: post #185
mattalex wrote: post #101

Earlier quoted context omitted.

You assume that openai's inference is profitable and that they aren't just trying to bolster revenue before their IPO. The only indication that openai is profitable comes from openai (whom I wouldn't trust with any statement, especially when it comes to profitability). In fact there is evidence that inference is not profitable simply because the rate of losses doesn't seem to reduce as revenue increases: if inference…

The indication that OpenAI's inference is profitable is that 3rd party providers host large models for cheaper. Given that OpenAI is ahead in intelligence, it's also reasonably likely that they are at the frontier of efficiency too. Your "evidence" for OpenAI's inference not being profitable is apparently based on leaked financials supposedly showing growing losses for reasons entirely unknown. With their research, t…

> Given that OpenAI is ahead in intelligence, it's also reasonably likely that they are at the frontier of efficiency too.

Frontier labs have no incentive to be at the frontier of efficiency.

Claude still leads the pack in general intelligence yet has the worst efficiency by far.

Re: Qwen3.8-Flash-Next

#210

Post by OutlawHusbando »

andy99 wrote: post #58

> Qwen3.8-Flash-Next features a 125B-parameter main model, supplemented by an additional 51B N-gram embeddings, with 6B parameters activated per token. Didn’t see this mentioned yet. I wonder what this means for the effective size. It’s evidently ~176B paramètres, but how does that get quantized. A 4-bit quant under 100GB seems unlikely, I’m suspecting this won’t run in 128GB unified memory In principle I like the id…

People in my server are running it on Strix Halo 128GB using RoCmFP4 and reporting 35tok/s, without much optimization, with proper MTP, better kernel, expecting about 50-60tok/s.
Post reply on HN