Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Qwen3-Next

qwen.ai

Post reply on HN
Print view View original

231–240 of 240 posts

Re: Qwen3-Next

#231

Post by mft_ »

hadlock wrote: post #170
siliconc0w wrote: post #160

All these new datacenters are going to be a huge sunk cost. Why would you pay OpenAI when you can host your own hyper efficient Chinese model for like 90% less cost at 90% of the performance. At that is compared to today's subsidized pricing, which they can't keep up forever.

Eventually Nvidia or a shrewd competitor will release 64/128gb consumer cards; locally hosted GPT 3.5+ is right around the corner, we're just waiting for consumer hardware to catch up at this point.

I think we're still at least an order of magnitude away (in terms of affordable local inference, or model improvements to squeeze more from less, or a combination of the two) from local solutions being seriously competitive for general purpose tasks, sadly.

I recently bought a second-hand 64GB Mac to experiment with. Even with the biggest recent local model it can run (llama3.3:70b just about runs acceptably; I've also tried an array of Qwen3 30b variants) the quality is lacking for coding support. They can sometimes write and iterate on a simple Python script, but sometimes fail, and for general-purpose models, often fail to answer questions accurately (not unsurprisingly, considering the model is a compression of knowledge, and these are comparatively small models). They are far, far away from the quality and ability of currently available Claude/Gemini/ChatGPT models. And even with a good eBay deal, the Mac cost the current equivalent of ~6 years of a monthly subscription to one of these.

Based on the current state of play, once we can access relatively affordable systems with 512-1024GB fast (v)ram and sufficient FLOPs to match, we might have a meaningfully powerful local solution. Until then, I fear local only is for enthusiasts/hobbyists and niche non-general tasks.

Re: Qwen3-Next

#232

Post by jbellis »

noahbp wrote: post #207
jbellis wrote: post #205

Added Qwen3 Next to the Brokk Power Ranking Open Round (coding benchmark). It's roughly GPT-OSS-20b strength. Full set of open weight model results: https://brokk.ai/power-ranking?version=openround&models=ds-r...

Is that the updated Kimi K2, or the old Kimi k2?

It's the original. I'll update the label to clarify.

Re: Qwen3-Next

#233

Post by mdp2021 »

dingnuts wrote: post #211
mdp2021 wrote: post #104

Earlier quoted context omitted.

> That's how LLMs work And that is also exactly how we want them not to work: we want them to be able to solve new problems. (Because Pandora's box is open, and they are not sold as a flexible query machine.) "Where was Napoleon born": easy. "How to resolve the conflict effectively": hard. Solved problems are interesting to students. Professionals have to deal with non trivial ones.

> how we want them not to work speak for yourself, I like solving problems and I'd like to retire before physical labor becomes the only way to support yourself > they are not sold as a flexible query machine yeah, SamA is a big fucking liar

I get your fear, d., but I am afraid we urgently need them tools, and to work properly. At some point in time the gap between workforce and objectives forced us to adopt cranes; at this point in time I see that "the carbon" is not "competing" enough. An IQ boost in the toolbox, when we will finally reach it, will be an enabler: for doom in the hands of fools, for the best in the hands of the wise - proportions worrisome but the game is not decided.

Meanwhile, there is no turning back and as the mockery of intelligence was invented, the Real Thing must be urgently found.

Edit: I have just read the title "Amateurish plan exposed failing diplomacy". The giants' list includes McNamara, Kissinger, Brzezinski: if some say that their efforts have not been sufficient - and failures are very costly -, what do we need?

Re: Qwen3-Next

#234

Post by cubefox »

Zacharias030 wrote: post #227
cubefox wrote: post #137

Earlier quoted context omitted.

> What kind of benefit does Multi-Token Prediction bring to the inference side? Is it only relevant in pretraining efficiency? It is only useful for inference and doesn't help with pretraining. Which actually points to speculative decoding not being sufficiently general, as the same underlying property (some sequences of tokens are easy to predict) could be exploited for training as well. See here: https://goombalab.…

There is no reason that it couldn’t be beneficial for training though.

Except that speculative decoding is de facto only an inference time optimization. But the H-Net architecture from the previous reference, which doesn't require tokens or speculative decoding, does something similar both for inference and training.

Re: Qwen3-Next

#235

Post by bigyabai »

jwr wrote: post #230
bigyabai wrote: post #172

Earlier quoted context omitted.

The model is 80b parameters, but only 3b are activated during inference. I'm running the old 2507 Qwen3 30B model on my 8gb Nvidia card and get very usable performance.

I understand that, but whether it's usable depends on whether ollama can load parts of it into memory on my Mac, and how quickly.

I really do not suggest ollama. It is slow, missing tons of llama.cpp features and doesn't expose many settings to the user. Koboldcpp is a much better inference provider and even has an ollama-compatible API endpoint.

Re: Qwen3-Next

#236

Post by Zacharias030 »

cubefox wrote: post #234
Zacharias030 wrote: post #227

Earlier quoted context omitted.

There is no reason that it couldn’t be beneficial for training though.

Except that speculative decoding is de facto only an inference time optimization. But the H-Net architecture from the previous reference, which doesn't require tokens or speculative decoding, does something similar both for inference and training.

Yes, but the discussion is about Multi-Token Prediction (Gloeckle et al. 2024) which is only incidentally useful for speculative decoding.

Re: Qwen3-Next

#237

Post by DiabloD3 »

EnPissant wrote: post #225
DiabloD3 wrote: post #224

Earlier quoted context omitted.

I suggest figuring out what your configuration problem is. Which llama.cpp flags are you using, because I am absolutely not having the same bug you are.

It's not a bug. It's the reality of token generation. It's bottlenecked by memory bandwidth. Please publish your own benchmarks proving me wrong.

I cannot reproduce your bug on AMD. I'm going to have to conclude this is a vendor issue.

Re: Qwen3-Next

#239

Post by jychang »

tomp wrote: post #144
stingraycharles wrote: post #108

Earlier quoted context omitted.

Because then the second token only needs to be checked, not generated, as it’s already generated? And it’s much faster to generate multiple tokens at the same time than one at a time? Is that the idea? I’m not an expert on LLMs, just a user.

No, the parent is wrong. Checking a token is the same as generating it. The benefit however is in the next (third) token. After generating tokens 1 and 2 (in one turn), you start generating token 3 (and 4). You also get the “real” prediction for token 2. If the “real” prediction matches the MTP (Multi-Token Prediction) from previous turn, you have just generated 3 correct tokens (and another speculative). If not, you…

To clarify, I should have stated: "Instead of generating tokens one at a time, you generate the second one as well WITH MTP, and then use speculative decoding on that second token (instead of having the second token be produced by a draft model like Qwen 0.6b). If the FIRST MTP token is checked and is correct, then the second token gets generated MUCH faster."

Re: Qwen3-Next

#240

Post by hadlock »

mft_ wrote: post #231
hadlock wrote: post #170

Earlier quoted context omitted.

Eventually Nvidia or a shrewd competitor will release 64/128gb consumer cards; locally hosted GPT 3.5+ is right around the corner, we're just waiting for consumer hardware to catch up at this point.

I think we're still at least an order of magnitude away (in terms of affordable local inference, or model improvements to squeeze more from less, or a combination of the two) from local solutions being seriously competitive for general purpose tasks, sadly. I recently bought a second-hand 64GB Mac to experiment with. Even with the biggest recent local model it can run (llama3.3:70b just about runs acceptably; I've al…

It would not surprise me at all to see 512, 768, 1024 gb models targeted at commercial or home users in the next 5 years. I can imagine a lot of companies, regulated ones in particular like finance, defense, medical, wanting to run the models in house, inside their own datacenter. A single card or pair of cards would probably be more than adequate for a thousand or more users, or half a dozen developers. If you already have a $25,000 database server, $12,000 for an "ai server" isn't a wild ask.
Post reply on HN