Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Qwen3-Next

qwen.ai

Post reply on HN
Print view View original

211–220 of 240 posts

Re: Qwen3-Next

#211

Post by dingnuts »

mdp2021 wrote: post #104
irthomasthomas wrote: post #31

Earlier quoted context omitted.

Naturally. That's how LLMs work. During training you measure the loss, the difference between the model output and the ground-truth and try to minimize it. We prize models for their ability to learn. Here we can see that the large model does a great job at learning to draw bob, while the small model performs poorly.

> That's how LLMs work And that is also exactly how we want them not to work: we want them to be able to solve new problems. (Because Pandora's box is open, and they are not sold as a flexible query machine.) "Where was Napoleon born": easy. "How to resolve the conflict effectively": hard. Solved problems are interesting to students. Professionals have to deal with non trivial ones.

> how we want them not to work

speak for yourself, I like solving problems and I'd like to retire before physical labor becomes the only way to support yourself

> they are not sold as a flexible query machine

yeah, SamA is a big fucking liar

Re: Qwen3-Next

#212

Post by ukuina »

sipjca wrote: post #194
jjcm wrote: post #192

Earlier quoted context omitted.

As plenty of others have mentioned here, if inference were 100x cheaper, I would run 200x inference. There are so many things you can do with long running, continuous inference.

but what if you don't need to run it in the cloud

You will ALWAYS want to use the absolute best model, because your time is more valuable than the machine's. If the machine gets faster or more capable, your value has jumped proportionally.

Re: Qwen3-Next

#213

Post by SparkyMcUnicorn »

jbellis wrote: post #205

Added Qwen3 Next to the Brokk Power Ranking Open Round (coding benchmark). It's roughly GPT-OSS-20b strength. Full set of open weight model results: https://brokk.ai/power-ranking?version=openround&models=ds-r...

This would be a valuable benchmark if it included languages other than Java, and let me see which models are best at the languages I work with.

My real-world usage does not line up with these results, but I'm not working with Java.

Re: Qwen3-Next

#214

Post by johntash »

NitpickLawyer wrote: post #24
yekanchi wrote: post #18

how much vram it requires?

A good rule of thumb is to think that one param is one unit of storage. The "default" unit of storage these days is bf16 (i.e. 16 bits for 1 weight). So for a 80B model that'll be ~160GB of weights. Then you have quantisation, usually in 8bit and 4bit. That means each weight is "stored" in 8bits or 4bits. So for a 80B model that'll be ~80GB in fp8 and ~40GB in fp4/int4. But in practice you need a bit more than that.…

Can you explain how context fits into this picture by any chance? I sort of understand the vram requirement for the model itself, but it seems like larger context windows increases the ram requirement by a lot more?

Re: Qwen3-Next

#215

Post by EnPissant »

furyofantares wrote: post #193
EnPissant wrote: post #183

Earlier quoted context omitted.

You don't. You run some of the layers on the CPU.

You're right that I was confused about that. LM Studio defaults to 12/36 layers on the GPU for that model on my machine, but you can crank it to all 36 on the GPU. That does slow it down but I'm not finding it unusable and it seems like it has some advantages - but I doubt I'm going to run it this way.

FWIW, that's a 80GB model and you also need kv cache. You'd need 96GBish to run on the GPU.

Re: Qwen3-Next

#216

Post by furyofantares »

EnPissant wrote: post #215
furyofantares wrote: post #193

Earlier quoted context omitted.

You're right that I was confused about that. LM Studio defaults to 12/36 layers on the GPU for that model on my machine, but you can crank it to all 36 on the GPU. That does slow it down but I'm not finding it unusable and it seems like it has some advantages - but I doubt I'm going to run it this way.

FWIW, that's a 80GB model and you also need kv cache. You'd need 96GBish to run on the GPU.

Do you know if it's doing what was described earlier, when I run it with all layers on GPU - paging an expert in every time the expert changes? Each expert is only 5.1B parameters.

Re: Qwen3-Next

#218

Post by EnPissant »

furyofantares wrote: post #216
EnPissant wrote: post #215

Earlier quoted context omitted.

FWIW, that's a 80GB model and you also need kv cache. You'd need 96GBish to run on the GPU.

Do you know if it's doing what was described earlier, when I run it with all layers on GPU - paging an expert in every time the expert changes? Each expert is only 5.1B parameters.

It makes absolutely no sense to do what OP described. The decode stage is bottlenecked on memory bandwidth. Once you pull the weights from system RAM, your work is almost done. To then gigabytes of weights PER TOKEN over PCIE to do some trivial computation on the GPU is crazy.

What actually happens is you run some or all of the MoE layers on the CPU from system RAM. This can be tolerable for smaller MoE models, but keeping it all on the GPU will still be 5-10x faster.

I'm guessing lmstudio gracefully falls back to running _soemthing_ on the CPU. Hopefully you are running only MoE on the CPU. I've only ever used llama.cpp.

Re: Qwen3-Next

#219

Post by lend000 »

rmuratov wrote: post #197
lend000 wrote: post #191

Prediction: AI will become commoditized ~15 IQ points higher than the state of the art models today, and with larger context, within 4 years as the incremental improvements in training from synthetic data plateaus (we've already used all the "real" data out there) and open source models are cheaply trained on the outputs of the big money models. Then AI development stagnates until someone invents an effective way to…

How did we use "all the data"? New knowledge appears on the internet every day, new scientific articles and videos are published.

At the speeds AI is moving, we've effectively used it all; the high quality data you need to make smarter models is coming in at a trickle. We're not getting 10^5 Principia Mathematicas published every day. Maybe I just don't have the vision to understand it, but it seems like AI-generated synthetic data for training shouldn't be able to make a smarter model than whatever produced that data. I can imagine synthetic data would be useful for making models more efficient (that's what quantized models are, after all), but not pushing the frontier.

Re: Qwen3-Next

#220

Post by lend000 »

HardCodedBias wrote: post #198
lend000 wrote: post #191

Prediction: AI will become commoditized ~15 IQ points higher than the state of the art models today, and with larger context, within 4 years as the incremental improvements in training from synthetic data plateaus (we've already used all the "real" data out there) and open source models are cheaply trained on the outputs of the big money models. Then AI development stagnates until someone invents an effective way to…

If that's true and if today's frontier models are around 120 IQ (who knows if that is true, but let's run with it, source: https://www.trackingai.org/home ) then we'll have an enormous number of ~135 IQ bots with nearly unlimited conscientiousness. I can't even begin to understand what that would mean.

Very interesting time to be alive.
Post reply on HN