These guys are setting up to absolutely own the global south market for AI. Which is in line with the belt and road initiative.
Qwen3-Coder-Next
311–320 of 443 posts
Re: Qwen3-Coder-Next
#312This GGUF is 48.4GB - https://huggingface.co/Qwen/Qwen3-Coder-Next-GGUF/tree/main/... - which should be usable on higher end laptops. I still haven't experienced a local model that fits on my 64GB MacBook Pro and can run a coding agent like Codex CLI or Claude code well enough to be useful. Maybe this will be the one? This Unsloth guide from a sibling comment suggests it might be: https://unsloth.ai/docs/models/qwen3…
you do realize claude opus/gpt5 are probably like 1000B-2000B models? So trying to have a model that's < 60B offer the same level of performance will be a miracle...
There is (must be - information theory) a size/capacity efficiency frontier. There is no particular reason to think we're anywhere near it right now.
Re: Qwen3-Coder-Next
#313Earlier quoted context omitted.
I don't buy this. I've long wondered if the larger models, while exhibiting more useful knowledge, are not more wasteful as we greedily explore the frontier of "bigger is getting us better results, make it bigger". Qwen3-Coder-Next seems to be a point for that thought: we need to spend some time exploring what smaller models are capable of. Perhaps I'm grossly wrong -- I guess time will tell.
You are not wrong, small models can be trained for niche use cases and there are lots of people and companies doing that. The problem is that you need one of those for each use case whereas the bigger models can cover a bigger problem space. There is also the counter-intuitive phenomenon where training a model on a wider variety of content than apparently necessary for the task makes it better somehow. For example, m…
Not an AI researcher and I don't really know, but intuitively it makes a lot of sense to me.
To do well as an LLM you want to end up with the weights that gets furthest in the direction of "reasoning".
So assume that with just one language there's a possibility to get stuck in local optima of weights that do well on the English test set but which doesn't reason well.
If you then take the same model size but it has to manage to learn several languages, with the same number of weights, this would eliminate a lot of those local optima because if you don't manage to get the weights into a regime where real reasoning/deeper concepts is "understood" then it's not possible to do well with several languages with the same number of weights.
And if you speak several languages that would naturally bring in more abstraction, that the concept of "cat" is different from the word "cat" in a given language, and so on.
Re: Qwen3-Coder-Next
#314Benchmarks using DGX Spark on vLLM 0.15.1.dev0+gf17644344
FP8: https://huggingface.co/Qwen/Qwen3-Coder-Next-FP8
Sequential (single request)
Prompt Gen Prompt Processing Token Gen
Tokens Tokens (tokens/sec) (tokens/sec)
------ ------ ----------------- -----------
521 49 3,157 44.2
1,033 83 3,917 43.7
2,057 77 3,937 43.6
4,105 77 4,453 43.2
8,201 77 4,710 42.2
Parallel (concurrent requests)
pp4096+tg128 (4K context, 128 gen):
n t/s
-- ----
1 28.5
2 39.0
4 50.4
8 57.5
16 61.4
32 62.0
pp8192+tg128 (8K context, 128 gen):
n t/s
-- ----
1 21.6
2 27.1
4 31.9
8 32.7
16 33.7
32 31.7Re: Qwen3-Coder-Next
#315Earlier quoted context omitted.
You can run plenty of models on a $10K machine or even a lot less than that, it all depends how much you want to wait for results. Streaming weights from SSD storage using mmap() is already a reality when running the largest and sparsest models. You can save even more on memory by limiting KV caching at the cost of extra compute, and there may be ways to push RAM savings even higher simply by tweaking the extent to w…
Yeah there's a lot of people that advocate for really slow inference on cheap infra. That's something else that should be expressed in this fidelity Because honestly I don't care about 0.2 tps for my use cases although I've spoken with many who are fine with numbers like that. At least the people I've talked to they talk about how if they have a very high confidence score that the model will succeed they don't mind t…
If you launch enough tasks in parallel you aren't going to care that 1 in 10 failed, as long as the other 9 are good. Just rerun the failed job whenever you get around to it, the infra will still be getting plenty of utilization on the rest.
Re: Qwen3-Coder-Next
#316Is it censored according to the wishes of the CCP?
Re: Qwen3-Coder-Next
#317This GGUF is 48.4GB - https://huggingface.co/Qwen/Qwen3-Coder-Next-GGUF/tree/main/... - which should be usable on higher end laptops. I still haven't experienced a local model that fits on my 64GB MacBook Pro and can run a coding agent like Codex CLI or Claude code well enough to be useful. Maybe this will be the one? This Unsloth guide from a sibling comment suggests it might be: https://unsloth.ai/docs/models/qwen3…
I can't get Codex CLI or Claude Code to use small local models and to use tools. This is because those tools use XML and the small local models have JSON tool use baked into them. No amount of prompting can fix it. In a day or two I'll release my answer to this problem. But, I'm curious, have you had a different experience where tool use works in one of these CLIs with a small local model?
I'm using this model right now in claude code with LM Studio perfectly, on a macbook pro
Re: Qwen3-Coder-Next
#318Earlier quoted context omitted.
If it sounds too good to be true…
Should be possible with optimised models, just drop all "generic" stuff and focus on coding performance. There's no reason for a coding model to contain all of ao3 and wikipedia =)
Now I wonder how strong the correlation between coding performance and ao3 knowledge is in human programmers. Maybe we are on to something here /s
Re: Qwen3-Coder-Next
#319Earlier quoted context omitted.
I can't get Codex CLI or Claude Code to use small local models and to use tools. This is because those tools use XML and the small local models have JSON tool use baked into them. No amount of prompting can fix it. In a day or two I'll release my answer to this problem. But, I'm curious, have you had a different experience where tool use works in one of these CLIs with a small local model?
I'm using this model right now in claude code with LM Studio perfectly, on a macbook pro
You mean Qwen3-Coder-Next? I haven't tried that model itself, yet, because I assume it's too big for me. I have a modest 16GB MacBook Air so I'm restricted to really small stuff. I'm thinking about buying a machine with a GPU to run some of these.
Anywayz, maybe I should try some other models. The ones that haven't worked for tool calling, for me are:
Llama3.1
Llama3.2
Qwen2.5-coder
Qwen3-coder
All these in 7b, 8b, or sometimes 30b (painfully) models.
I should also note that I'm typically using Ollama. Maybe LM Studio or llama.cpp somehow improve on this?
Re: Qwen3-Coder-Next
#320Earlier quoted context omitted.
We need a new word, not "local model" but "my own computers model" CapEx based This distinction is important because some "we support local model" tools have things like ollama orchestration or use the llama.cpp libraries to connect to models on the same physical machine. That's not my definition of local. Mine is "local network". so call it the "LAN model" until we come up with something better. "Self-host" exists b…
I don't even need "open weights" to run on hardware I own. I am fine renting an H100 (or whatever), as long as I theoretically have access to and own everything running. I do not want my career to become dependent upon Anthropic. Honestly, the best thing for "open" might be for us to build open pipes and services and models where we can rent cloud. Large models will outpace small models: LLMs, video models, "world" m…
> I do not want my career to become dependent upon Anthropic
As someone who switches between Anthropic and ChatGPT depending on the month and has dabbled with other providers and some local LLMs, I think this fear is unfounded.
It's really easy to switch between models. The different models have some differences that you notice over time but the techniques you learn in one place aren't going to lock you into a provider anywhere.