Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Qwen3-Next

qwen.ai

Post reply on HN
Print view View original

181–190 of 240 posts

Re: Qwen3-Next

#181

Post by siliconc0w »

GaggiX wrote: post #168
siliconc0w wrote: post #160

All these new datacenters are going to be a huge sunk cost. Why would you pay OpenAI when you can host your own hyper efficient Chinese model for like 90% less cost at 90% of the performance. At that is compared to today's subsidized pricing, which they can't keep up forever.

>to today's subsidized pricing, which they can't keep up forever. The APIs are not subsidized, they probably have quite the large margin actually: https://lmsys.org/blog/2025-05-05-large-scale-ep/ >Why would you pay OpenAI when you can host your own hyper efficient Chinese model The 48GB of VRAM or unified memory required to run this model at 4bits is not free either.

I didn't say its free but it is about 90% cheaper. Sonnet is $15 per million token output, this just dropped and is available at OpenRouter at $1.40. Even compared to Gemini Flash which is probably the best price-to-performance API is generally ranked lower than Qwen's models and is $2.50 so still %44 cheaper.

Re: Qwen3-Next

#182

Post by EnPissant »

zettabomb wrote: post #112
EnPissant wrote: post #41

Earlier quoted context omitted.

What you are describing would be uselessly slow and nobody does that.

llama.cpp has built-in support for doing this, and it works quite well. Lots of people running LLMs on limited local hardware use it.

llama.cpp has support for running some of or all of the layers on the CPU. It does not swap them into the GPU as needed.

Re: Qwen3-Next

#184

Post by EnPissant »

DiabloD3 wrote: post #56
EnPissant wrote: post #41

Earlier quoted context omitted.

What you are describing would be uselessly slow and nobody does that.

I don't load all the MoE layers onto my GPU, and I have only about a 15% reduction in token generation speed while maintaining a model 2-3 times larger than VRAM alone.

The slowdown is far more than 15% for token generation. Token generation is mostly bottlenecked by memory bandwidth. Dual channel DDR5-6000 has 96GB/s and A rtx 5090 has 1.8TB/s. See my other comment when I show 5x slowdown in token generation by moving just the experts to the CPU.

Re: Qwen3-Next

#187

Post by theptip »

mmmllm wrote: post #82
amelius wrote: post #70

Earlier quoted context omitted.

https://en.wikipedia.org/wiki/Jevons_paradox

Sure but where is the demand going to come from? LLMs are already in every google search, in Whatsapp/Messenger, throughout Google workspace, Notion, Slack, etc. ChatGPT already has a billion users. Plus penetration is already very high in the areas where they are objectively useful: programming, customer care etc. I just don't see where the 100-1000x demand comes from to offset this. Would be happy to hear other vie…

I’m already throughput-capped on my output via Claude. If you gave me 10x the token/s I’d ship at least twice as much value (at good-enough for the business quality, to be clear).

There are plenty of usecases where the models are not smart enough to solve the problem yet, but there is very obviously a lot of value available to be harvested from maturing and scaling out just the models we already have.

Concretely, the $200/mo and $2k/ mo offerings will be adopted by more prosumer and professional users as the product experience becomes more mature.

Re: Qwen3-Next

#188

Post by KaoruAoiShiho »

jug wrote: post #177
dragonwriter wrote: post #157

Earlier quoted context omitted.

> I'm skeptical about these claims. How can this be? More efficient architecture. > Wouldn't there be massive loss of world knowledge? If you assume equally efficient architecture and no other salient differences, yes, that’s what you’d expect from a smaller model.

Hmm. Let's just say if this is true, that this is actually better with such a much lower total parameter count, it's the greatest accomplishment in over a year of LLM development. With the backdrop of bechmaxxing in 2025, I'll believe in this when I see the results on closed benchmarks and SimpleBench. My concern is this might be a hallucination machine.

In my testing this model is quite bad and far behind 235b a22b. https://fiction.live/stories/Fiction-liveBench-Sept-12-2025/...

Re: Qwen3-Next

#189

Post by EnPissant »

bigyabai wrote: post #173
EnPissant wrote: post #41

Earlier quoted context omitted.

What you are describing would be uselessly slow and nobody does that.

I run the 30B Qwen3 on my 8GB Nvidia GPU and get a shockingly high tok/s.

For contrast, I get the following for a rtx 5090 and 30b qwen3 coder quantized to ~4 bits:

- Prompt processing 65k tokens: 4818 tokens/s

- Token generation 8k tokens: 221 tokens/s

If I offload just the experts to run on the CPU I get:

- Prompt processing 65k tokens: 3039 tokens/s

- Token generation 8k tokens: 42.85 tokens/s

As you can see, token generation is over 5x slower. This is only using ~5.5GB VRAM, so the token generation could be sped up a small amount by moving a few of the experts onto the GPU.

Re: Qwen3-Next

#190

Post by porridgeraisin »

pmarreck wrote: post #150
porridgeraisin wrote: post #113

Earlier quoted context omitted.

Background: LLMs take your input, upscale it into a very high dimensional space, and then downscale it back to 1D at the end. This 1D list is interpreted as a list of probabilities -- one for each word in your vocabulary. i.e f(x) = downscale(upscale(x)). Each of downscale() and upscale() are parameterized (billions of params). I see you have a gamedev background, so as an example: bezier curves are parameterized fun…

so after your edit it would be (just to clarify): I use ____ ___ = downscale_common(lightweight1(.)) + downscale_common(lightweight2(.)) ? And does it generate 2 at a time and keep going that way, or is there some overlap?

You generate blocks of 2 at a time yes. In general, k. As you can imagine, larger k performs worse. LLM(I like cats) is very likely to continue with "because they", but beyond that, there's too many possibilities. LLM(I like cats because they are) = small and cute and they meow, while LLM(I like cats because they eat) = all the rats in my garden.

If you try to predict the whole thing at once you might end up with

I like cats because they are all the rats and they garden

> Overlap

Check out an inference method called self-speculative decoding which solves(somewhat) the above problem of k-token prediction, which does overlap the same ___ across multiple computations.

Post reply on HN