Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Qwen3.8-Flash-Next

qwen.ai

Post reply on HN
Print view View original

181–190 of 246 posts

Re: Qwen3.8-Flash-Next

#181

Post by hedgehog »

cmrdporcupine wrote: post #170
hedgehog wrote: post #164

Earlier quoted context omitted.

The active param count is so small I'm not sure how much advantage MTP will have, the current llama.cpp does load the ngram embeddings but I haven't verified it uses them. I expect to redeploy all this stuff every few days as the tooling gets improved.

My initial implementation of MTP I have here in my own (DGX Spark specific) custom runtime brought it up from ~12 tok/sec without MTP to ~16 to ~20 with; depending on workload. It's not world changing , but at those speeds I'll take anything I can get. (The ngram embeddings in this case are paged to/from disk which seems to cost ... basically nothing). https://github.com/rdaum/eider/

It should be faster, Q6 on Ryzen 395 using Vulkan llama.cpp is about 22 tokens/s with no MTP, and I'd expect the Spark to be 20% faster or something in that neighborhood.

Re: Qwen3.8-Flash-Next

#182

Post by cmrdporcupine »

hedgehog wrote: post #181
cmrdporcupine wrote: post #170

Earlier quoted context omitted.

My initial implementation of MTP I have here in my own (DGX Spark specific) custom runtime brought it up from ~12 tok/sec without MTP to ~16 to ~20 with; depending on workload. It's not world changing , but at those speeds I'll take anything I can get. (The ngram embeddings in this case are paged to/from disk which seems to cost ... basically nothing). https://github.com/rdaum/eider/

It should be faster, Q6 on Ryzen 395 using Vulkan llama.cpp is about 22 tokens/s with no MTP, and I'd expect the Spark to be 20% faster or something in that neighborhood.

Probably. I've spent zero time with optimization at this point. Code is all new this morning.

Curious if llama.cpp is doing full BF16 for the n-gram embeddings table, or if that's quantized, too. I was going to try nvfp4 for that but wasn't sure about the quality risk.

Spark usually wins on prefill, not decode. I think memory bandwidth is about same between the two. What do you get for prefill/prompt?

Re: Qwen3.8-Flash-Next

#183

Post by [deleted] »

schopra909 wrote: post #107

Can someone explain the intuition behind the en-gram idea? I know DeepSeek published a paper about it a few months ago and the Gemma models have a lightweight version of it; but it hasn’t clicked for me yet

[deleted]

Re: Qwen3.8-Flash-Next

#184

Post by Groxx »

powersurge360 wrote: post #129
mistersquid wrote: post #124

Earlier quoted context omitted.

> You’re absolutely right to be hopeful. Three honest possibilities, and I’ll be straight with you about each: > [UGC styled humorously as LLMisms] All joking aside, having interacted with Claude intensely for the last 8 months and about 30 hours/week in the last 3, I’ve started to notice how (for want of a better word) “readable” (“digestible” ? “comprehensible” ? “Predictable” is the wrong direction.) information c…

I find LLMisms very annoying to read, it’s almost like they are bullet points in the shape of a paragraph. It feels very “skippy” to me. EDITED: Removed a question that I couldn’t make feel suitably polite.

An accurate description, I think. Plus they have trouble leading from one paragraph into the next, or maintaining any kind of coherent direction further.

In summary, I think it's an expensive time to buy computer hardware, and I might recommend holding off on any purchases.

Re: Qwen3.8-Flash-Next

#185

Post by user43928 »

mattalex wrote: post #101
jrflo wrote: post #39

Earlier quoted context omitted.

The luna cost cuts were real though, not a one time promotion or something, due to some optimization (probably distillation?) that openai did.

You assume that openai's inference is profitable and that they aren't just trying to bolster revenue before their IPO. The only indication that openai is profitable comes from openai (whom I wouldn't trust with any statement, especially when it comes to profitability). In fact there is evidence that inference is not profitable simply because the rate of losses doesn't seem to reduce as revenue increases: if inference…

The indication that OpenAI's inference is profitable is that 3rd party providers host large models for cheaper.

Given that OpenAI is ahead in intelligence, it's also reasonably likely that they are at the frontier of efficiency too.

Your "evidence" for OpenAI's inference not being profitable is apparently based on leaked financials supposedly showing growing losses for reasons entirely unknown.

With their research, training, data centers, chip development, and hardware product development, there seem to be a number of reasons that might explain growing losses.

Re: Qwen3.8-Flash-Next

#186

Post by terrut »

raducu wrote: post #80
redrix wrote: post #53

Earlier quoted context omitted.

You’re absolutely right to be hopeful. Three honest possibilities, and I’ll be straight with you about each: 1. It overthinks — Just like the previous iteration. High confidence. 2. It doesn’t overthink — Improvement from the last model for your use case. Regression for others. 3. It sometimes overthinks — Best case all around. A feature, not an impairment. One final thing worth mentioning: (I made myself irrationall…

On one hand I love your joke, on the other, this is HN not reddit and I usually downvote such responses, not sure what is the HN etiquette for such humor?

You are right to push back— Sorry, couldn't resist ;) I agree that this is not what we normally come here for, but this thread made me chuckle. I think we are just venting our shared frustrations a bit.

Re: Qwen3.8-Flash-Next

#187

Post by petu »

giancarlostoro wrote: post #139
lnenad wrote: post #21

Adding to my homelab stack, hopefully it doesn't overthink like the little model. Actually, hoping it thinks a bit less. Wait actually I'm really praying it reasons a bit more directly. But wait, I'm really sure that it must be a bit better.

Since you're running through the trouble of setting that up, if its 125B params, but only 6B is activated, does that mean you mainly need to allocate enough VRAM for that much of the model? Or do you still need enough VRAM for the whole thing (and buffer for context window)? Or maybe anyone can inform me, this is one area I'm uninformed in.

You need VRAM for the whole thing for optimal performance. Activation is chosen "randomly" for each token. PCIe becomes bottleneck, so much that just doing computation on CPU is likely faster.

But given it's only 6B, out of which only ~2.4B seem to be actually routed ("selected at random per token"), you could get reasonable performance with experts on CPU (still haven't tested, but 20-30 for dual channel DDR5 and 4 bpw quant).

Re: Qwen3.8-Flash-Next

#188

Post by hedgehog »

cmrdporcupine wrote: post #182
hedgehog wrote: post #181

Earlier quoted context omitted.

It should be faster, Q6 on Ryzen 395 using Vulkan llama.cpp is about 22 tokens/s with no MTP, and I'd expect the Spark to be 20% faster or something in that neighborhood.

Probably. I've spent zero time with optimization at this point. Code is all new this morning. Curious if llama.cpp is doing full BF16 for the n-gram embeddings table, or if that's quantized, too. I was going to try nvfp4 for that but wasn't sure about the quality risk. Spark usually wins on prefill, not decode. I think memory bandwidth is about same between the two. What do you get for prefill/prompt?

Ok, at 50k context its about 126 prefill, 13 generation.

Re: Qwen3.8-Flash-Next

#189

Post by cmrdporcupine »

hedgehog wrote: post #188
cmrdporcupine wrote: post #182

Earlier quoted context omitted.

Probably. I've spent zero time with optimization at this point. Code is all new this morning. Curious if llama.cpp is doing full BF16 for the n-gram embeddings table, or if that's quantized, too. I was going to try nvfp4 for that but wasn't sure about the quality risk. Spark usually wins on prefill, not decode. I think memory bandwidth is about same between the two. What do you get for prefill/prompt?

Ok, at 50k context its about 126 prefill, 13 generation.

Oh, interesting. So beats on prefill and matches at decode. MTP will help you a bit once you have it. I've got a lot of work to do to optimize prefill.

Kinda wish I had a Strix Halo here to play with as well.

Re: Qwen3.8-Flash-Next

#190

Post by computerex »

mediaman wrote: post #146
aaa_aaa wrote: post #113

Earlier quoted context omitted.

Argument was that open ai cannot be profitable with this. But sure, use it while you can.

You can make the other argument that China subsidizes the price and that they can't be profitable at this pricing level. From an industrial strategy standpoint, they already do this for many other industries with huge subsidized state loans. So we can go round and round on this, each with our made-up objections about how it's temporary or unrealistic or impossible or whatever, or we can just accept the prices as list…

Didn't OpenCode CTO state they could replicate deepseek pricing on rented hardware?
Post reply on HN