So dang exciting! There are a bunch of new interesting small models out lately, by the way, this is just one of them...
Qwen3-Coder-Next
301–310 of 443 posts
Re: Qwen3-Coder-Next
#302Earlier quoted context omitted.
I wonder if the future in ~5 years is almost all local models? High-end computers and GPUs can already do it for decent models, but not sota models. 5 years is enough time to ramp up memory production, consumers to level-up their hardware, and models to optimize down to lower-end hardware while still being really good.
Opensource or local models will always heavily lag frontier. Who pays for a free model? GPU training isn't free! I remember early on people saying 100B+ models will run on your phone like nowish. They were completely wrong and I don't think it's going to ever really change. People always will want the fastest, best, easiest setup method. "Good enough" massively changes when your marketing team is managing k8s cluster…
I don't know about frontier, I code nowadays a lot using Opus 4.5, in a way that I instruct it to do something (like complex refactor etc) - I like that it's really good at actually doing what its told and only occasionally do I have to fight it when it goes off the rails. It also does not hallucinate all that much in my experience (Im writing Js, YMMV with other languages), and is good at spotting dumb mistakes.
That said, I'm not sure if this capability is only achievable in huge frontier models, I would be perfectly content using a model that can do this (acting as a force multiplier), and not much else.
Re: Qwen3-Coder-Next
#303Earlier quoted context omitted.
I wonder if the future in ~5 years is almost all local models? High-end computers and GPUs can already do it for decent models, but not sota models. 5 years is enough time to ramp up memory production, consumers to level-up their hardware, and models to optimize down to lower-end hardware while still being really good.
Opensource or local models will always heavily lag frontier. Who pays for a free model? GPU training isn't free! I remember early on people saying 100B+ models will run on your phone like nowish. They were completely wrong and I don't think it's going to ever really change. People always will want the fastest, best, easiest setup method. "Good enough" massively changes when your marketing team is managing k8s cluster…
The calculation will probably get better for locally hosted models once investor generosity runs out for the remotely hosted models.
Re: Qwen3-Coder-Next
#304I just tried qwen 3 tts and it was mind blowingly good, you can even provide directions for the overall tone etc. Which wasn't the case when I used commercial super expensive products like the (now closed after being bought by meta) play.ht .
Does anyone see a reason to still use elevenlabs etc. ?
Re: Qwen3-Coder-Next
#305As always, the Qwen team is pushing out fantastic content Hope they update the model page soon https://chat.qwen.ai/settings/model
> "content" Sorry, but we're talking about models as content now? There's almost always a better word than "content" if you're describing something that's in tech or online.
I wasn’t only referring to their new model, I meant their blogpost and the research behind their progress, its always a joyride to read.
I didn’t know it was this serious with the vocabulary, I’ll be more cautious in the future.
Re: Qwen3-Coder-Next
#306Earlier quoted context omitted.
You are not wrong, small models can be trained for niche use cases and there are lots of people and companies doing that. The problem is that you need one of those for each use case whereas the bigger models can cover a bigger problem space. There is also the counter-intuitive phenomenon where training a model on a wider variety of content than apparently necessary for the task makes it better somehow. For example, m…
Is that counterintuitive? If I had a model trained on 10 different programming languages, including my target language, I would expect it to do better than a model trained only on my target language, simply because it has access to so much more code/algorithms/examples then my language alone. i.e. there is a lot of commonality between programming languages just as there is between human languages, so training on one…
> simply because it has access to so much more code/algorithms/examples then my language alone
I assumed that is what was catered for with "even when controlling for the size of the training set".
I.e. assuming I am reading it right: That it is better to get the same data as 25% in 4 languages, than 100% in one language.
Re: Qwen3-Coder-Next
#307This GGUF is 48.4GB - https://huggingface.co/Qwen/Qwen3-Coder-Next-GGUF/tree/main/... - which should be usable on higher end laptops. I still haven't experienced a local model that fits on my 64GB MacBook Pro and can run a coding agent like Codex CLI or Claude code well enough to be useful. Maybe this will be the one? This Unsloth guide from a sibling comment suggests it might be: https://unsloth.ai/docs/models/qwen3…
I can't get Codex CLI or Claude Code to use small local models and to use tools. This is because those tools use XML and the small local models have JSON tool use baked into them. No amount of prompting can fix it. In a day or two I'll release my answer to this problem. But, I'm curious, have you had a different experience where tool use works in one of these CLIs with a small local model?
Surely the answer is a very small proxy server between the two?
Re: Qwen3-Coder-Next
#308Earlier quoted context omitted.
We need a new word, not "local model" but "my own computers model" CapEx based This distinction is important because some "we support local model" tools have things like ollama orchestration or use the llama.cpp libraries to connect to models on the same physical machine. That's not my definition of local. Mine is "local network". so call it the "LAN model" until we come up with something better. "Self-host" exists b…
You can run plenty of models on a $10K machine or even a lot less than that, it all depends how much you want to wait for results. Streaming weights from SSD storage using mmap() is already a reality when running the largest and sparsest models. You can save even more on memory by limiting KV caching at the cost of extra compute, and there may be ways to push RAM savings even higher simply by tweaking the extent to w…
Yeah there's a lot of people that advocate for really slow inference on cheap infra. That's something else that should be expressed in this fidelity
Because honestly I don't care about 0.2 tps for my use cases although I've spoken with many who are fine with numbers like that.
At least the people I've talked to they talk about how if they have a very high confidence score that the model will succeed they don't mind the wait.
Essentially a task failure is 1 in 10, I want to monitor and retry.
If it's 1 in 1000, then I can walk away.
The reality is most people don't have a bearing on what this order of magnitude actually is for a given task. So unless you have high confidence in your confidence score, slow is useless
But sometimes you do...
Re: Qwen3-Coder-Next
#309Earlier quoted context omitted.
I can't get Codex CLI or Claude Code to use small local models and to use tools. This is because those tools use XML and the small local models have JSON tool use baked into them. No amount of prompting can fix it. In a day or two I'll release my answer to this problem. But, I'm curious, have you had a different experience where tool use works in one of these CLIs with a small local model?
Surely the answer is a very small proxy server between the two?
That might work, but I keep seeing people talk about this, so there must be a simple solution that I'm over-looking. My solution is to write my own minimal and experimental CLI that talks JSON tools.
Re: Qwen3-Coder-Next
#310As always, the Qwen team is pushing out fantastic content Hope they update the model page soon https://chat.qwen.ai/settings/model
That’s a perfectly fine usage of content (primary substance offered by a “website”)