Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Qwen3-Coder-Next

qwen.ai

Post reply on HN
Print view View original

341–350 of 443 posts

Re: Qwen3-Coder-Next

#341

Post by echelon »

christkv wrote: post #338
airstrike wrote: post #325

Earlier quoted context omitted.

right, but ChatGPT might not exist at some point, and if we don't force feed the open inference ecosystem and infrastructure back into the mouths of the AI devourer that is this hype cycle, we'll simply be accepting our inevitable, painful death

If they die there will be so much hardware released to do other tasks.

Perhaps not tasks you get the opportunity to do.

Your job might be assigned to some other legal entity renting some other compute.

If this goes as according to some of their plans, we might all be out of the picture one day.

If these systems are closed, you might not get the opportunity to hire them yourself to build something you have ownership in. You might be cut out.

Re: Qwen3-Coder-Next

#342

Post by epolanski »

segmondy wrote: post #118
simonw wrote: post #3

This GGUF is 48.4GB - https://huggingface.co/Qwen/Qwen3-Coder-Next-GGUF/tree/main/... - which should be usable on higher end laptops. I still haven't experienced a local model that fits on my 64GB MacBook Pro and can run a coding agent like Codex CLI or Claude code well enough to be useful. Maybe this will be the one? This Unsloth guide from a sibling comment suggests it might be: https://unsloth.ai/docs/models/qwen3…

you do realize claude opus/gpt5 are probably like 1000B-2000B models? So trying to have a model that's < 60B offer the same level of performance will be a miracle...

Aren't both latest opus and sonnet smaller than the previous versions?

Re: Qwen3-Coder-Next

#343

Post by johnsmith1840 »

margalabargala wrote: post #258
johnsmith1840 wrote: post #203

Earlier quoted context omitted.

Opensource or local models will always heavily lag frontier. Who pays for a free model? GPU training isn't free! I remember early on people saying 100B+ models will run on your phone like nowish. They were completely wrong and I don't think it's going to ever really change. People always will want the fastest, best, easiest setup method. "Good enough" massively changes when your marketing team is managing k8s cluster…

I don't think this is as true as you think. People do not care about the fastest and best past a point. Let's use transportation as an analogy. If all you have is a horse, a car is a massive improvement. And when cars were just invented, a car with a 40mph top speed was a massive improvement over one with a 20mph top speed and everyone swapped. While cars with 200mph top speeds exist, most people don't buy them. We a…

This is the assumption of a hard plateu we can effectively optimize forever towards while possible we havn't seen it.

Again my point is "good enough" changes as possibilities open. Marketing teams running entire infra stacks is an insane idea today but may not be in the future.

You could easily code with a local model similar to gpt 4 or 3 now but I will 10-100x your performance with a frontier model and that will fundamentally not change.

Hmmm but maybe there's an argument of a static task. Once a model hits that ability of that specific task you can optimize it into a smaller model. So I guess I buy the argument for people working on statically capped conplexity tasks?

PII detection for example, a <500M model will outperform a 1-8B param model on that narrow task. But at the same time just a pii detection bot is not a product anymore. So yes a opensource one does it but as a result its fundamentally less valuable and I need to build higher and larger products for the value?

Re: Qwen3-Coder-Next

#344

Post by johnsmith1840 »

kybernetikos wrote: post #261
johnsmith1840 wrote: post #203

Earlier quoted context omitted.

Opensource or local models will always heavily lag frontier. Who pays for a free model? GPU training isn't free! I remember early on people saying 100B+ models will run on your phone like nowish. They were completely wrong and I don't think it's going to ever really change. People always will want the fastest, best, easiest setup method. "Good enough" massively changes when your marketing team is managing k8s cluster…

Gpt3.5 as used in the first commercially available chat gpt is believed to be hundreds of billions of parameters. There are now models I can run on my phone that feel like they have similar levels of capability. Phones are never going to run the largest models locally because they just don't have the size, but we're seeing improvements in capability at small sizes over time that mean that you can run a model on your…

Sure but the moment you can use that small model locally its capabilities are no longer differntiated or valuable no?

I supose the future will look exacrly like now. Some mixture of local and non local.

I guess my argument is that market dominated by local doesn't seem right and I think the balance will look similar to what it is right now

Re: Qwen3-Coder-Next

#345

Post by enlyth »

dehrmann wrote: post #47
simonw wrote: post #3

This GGUF is 48.4GB - https://huggingface.co/Qwen/Qwen3-Coder-Next-GGUF/tree/main/... - which should be usable on higher end laptops. I still haven't experienced a local model that fits on my 64GB MacBook Pro and can run a coding agent like Codex CLI or Claude code well enough to be useful. Maybe this will be the one? This Unsloth guide from a sibling comment suggests it might be: https://unsloth.ai/docs/models/qwen3…

I wonder if the future in ~5 years is almost all local models? High-end computers and GPUs can already do it for decent models, but not sota models. 5 years is enough time to ramp up memory production, consumers to level-up their hardware, and models to optimize down to lower-end hardware while still being really good.

I'm hoping so. What's amazing is that with local models you don't suffer from what I call "usage anxiety" where I find myself saving my Claude usage for hypothetical more important things that may come up, or constantly adjusting prompts and doing some manual work myself to spare token usage.

Having this power locally means you can play around and experiment more without worries, it sounds like a wonderful future.

Re: Qwen3-Coder-Next

#346

Post by taneq »

openclawai wrote: post #332
kristopolous wrote: post #255

Earlier quoted context omitted.

We need a new word, not "local model" but "my own computers model" CapEx based This distinction is important because some "we support local model" tools have things like ollama orchestration or use the llama.cpp libraries to connect to models on the same physical machine. That's not my definition of local. Mine is "local network". so call it the "LAN model" until we come up with something better. "Self-host" exists b…

For context on what cloud API costs look like when running coding agents: With Claude Sonnet at $3/$15 per 1M tokens, a typical agent loop with ~2K input tokens and ~500 output per call, 5 LLM calls per task, and 20% retry overhead (common with tool use): you're looking at roughly $0.05-0.10 per agent task. At 1K tasks/day that's ~$1.5K-3K/month in API spend. The retry overhead is where the real costs hide. Most cost…

At this point isn’t the marginal cost based on power consumption? At 30c/kWh and with a beefy desktop pc pulling up to half a kW, that’s 15c/hr. For true zero marginal cost, maybe get solar panels. :P

Re: Qwen3-Coder-Next

#347

Post by anon373839 »

cgearhart wrote: post #274
dust42 wrote: post #44

Earlier quoted context omitted.

Unfortunately Qwen3-next is not well supported on Apple silicon, it seems the Qwen team doesn't really care about Apple. On M1 64GB Q4KM on llama.cpp gives only 20Tok/s while on MLX it is more than twice as fast. However, MLX has problems with kv cache consistency and especially with branching. So while in theory it is twice as fast as llama.cpp it often does the PP all over again which completely trashes performance…

Any notes on the problems with MLX caching? I’ve experimented with local models on my MacBook and there’s usually a good speedup from MLX, but I wasn’t aware there’s an issue with prompt caching. Is it from MLX itself or LMstudio/mlx-lm/etc?

There’s this issue/outstanding PR: https://github.com/lmstudio-ai/mlx-engine/pull/188#issuecomm...

Re: Qwen3-Coder-Next

#348

Post by cmrdporcupine »

adefa wrote: post #314

Benchmarks using DGX Spark on vLLM 0.15.1.dev0+gf17644344 FP8: https://huggingface.co/Qwen/Qwen3-Coder-Next-FP8 Sequential (single request) Prompt Gen Prompt Processing Token Gen Tokens Tokens (tokens/sec) (tokens/sec) ------ ------ ----------------- ----------- 521 49 3,157 44.2 1,033 83 3,917 43.7 2,057 77 3,937 43.6 4,105 77 4,453 43.2 8,201 77 4,710 42.2 Parallel (concurrent requests) pp4096+tg128 (4K context, 12…

I tried the FP8 in vLLM on my Spark and although it fit in memory, I started swapping once I actually tried to run any queries, and, yeah, could not have a context larger than 8k.

I figured out later this is because vLLM apparently de-quantizes to BF16 at runtime, so pointless to run the FP8?

I get about 30-35 tok/second using llama.cpp and a 4-bit quant. And a 200+k context, using only 50GB of RAM.

Re: Qwen3-Coder-Next

#349

Post by MrDrMcCoy »

mittermayr wrote: post #223
organsnyder wrote: post #89

Earlier quoted context omitted.

They run fairly well for me on my 128GB Framework Desktop.

what do you run this on if I may ask? lmstudio, ollama, lama? which cli?

Can't speak for parent, but I've had decent luck with llama.cpp on my triple Ryzen AI Pro 9700 XTs.
Post reply on HN