Is there a good way to enable this model within VSCode, looking for something like Copilot?
Qwen3-Coder-Next
371–380 of 443 posts
Re: Qwen3-Coder-Next
#372Earlier quoted context omitted.
I tried the FP8 in vLLM on my Spark and although it fit in memory, I started swapping once I actually tried to run any queries, and, yeah, could not have a context larger than 8k. I figured out later this is because vLLM apparently de-quantizes to BF16 at runtime, so pointless to run the FP8? I get about 30-35 tok/second using llama.cpp and a 4-bit quant. And a 200+k context, using only 50GB of RAM.
Running llama.cpp rather than vLLM, it's happy enough to run the FP8 variant with 200k+ context using about 90GB vram
yeah, what did you get for tok/sec there though? Memory bandwidth is the limitation with these devices. With 4 bit I didn't get over 35-39 tok/sec, and averaged more like 30 when doing actual tool use with opencode. I can't imagine fp8 being faster.
Re: Qwen3-Coder-Next
#373Not crazy about it. It keeps getting stuck in a loop and filling up the context window (131k, run locally). Kimi's been nice, even if a bit slow.
Did you apply RoPE?
Re: Qwen3-Coder-Next
#374This is model 12188, which claims to rival SOTA models while not even being in the same league.
In terms of intelligence per compute, it’s probably the best model I can realistically run locally on my laptop for coding. It’s solid for scripting and small projects.
I tried it on a mid-size codebase (~50k LOC), and the context window filled up almost immediately, making it basically unusable unless you’re extremely explicit about which files to touch. I tested it with a 8k context window but will try again with 32k and see if it becomes more practical.
I think the main blocker for using local coding models more is the context window. A lot of work is going into making small models “smarter,” but for agentic coding that only gets you so far. No matter how smart the model is, an agent will blow through the context as soon as it reads a handful of files.
Re: Qwen3-Coder-Next
#375This is model 12188, which claims to rival SOTA models while not even being in the same league. In terms of intelligence per compute, it’s probably the best model I can realistically run locally on my laptop for coding. It’s solid for scripting and small projects. I tried it on a mid-size codebase (~50k LOC), and the context window filled up almost immediately, making it basically unusable unless you’re extremely exp…
The small context window has been a recognized problem for a while now. Really only Google has the ability to use a good long context window
Re: Qwen3-Coder-Next
#376Earlier quoted context omitted.
right, but ChatGPT might not exist at some point, and if we don't force feed the open inference ecosystem and infrastructure back into the mouths of the AI devourer that is this hype cycle, we'll simply be accepting our inevitable, painful death
> right, but ChatGPT might not exist at some point There are multiple frontier models to choose from. They’re not all going to disappear.
right, and the less we rely on ChatGPT and Claude, the more we give power to "all other frontier models", which right now have very, very little market share
Re: Qwen3-Coder-Next
#377Earlier quoted context omitted.
We need a new word, not "local model" but "my own computers model" CapEx based This distinction is important because some "we support local model" tools have things like ollama orchestration or use the llama.cpp libraries to connect to models on the same physical machine. That's not my definition of local. Mine is "local network". so call it the "LAN model" until we come up with something better. "Self-host" exists b…
For context on what cloud API costs look like when running coding agents: With Claude Sonnet at $3/$15 per 1M tokens, a typical agent loop with ~2K input tokens and ~500 output per call, 5 LLM calls per task, and 20% retry overhead (common with tool use): you're looking at roughly $0.05-0.10 per agent task. At 1K tasks/day that's ~$1.5K-3K/month in API spend. The retry overhead is where the real costs hide. Most cost…
On the other hand, Deepseek V3.2 is $0.38 per million tokens output. And on openrouter, most providers serve it at 20 tokens/sec.
At 20t/s over 1 month, that's... $19something running literally 24/7. In reality it'd be cheaper than that.
I bet you'd burn more than $20 in electricity with a beefy machine that can run Deepseek.
The economics of batch>1 inference does not go in favor of consumers.
Re: Qwen3-Coder-Next
#378Earlier quoted context omitted.
Did you have to do anything special to get it to work? I tried and it would just bug out, things like respond with JSON strings summarizing what I asked of it or just outright getting things wrong entirely. For example, I asked it to summarize what a specific .js file did and it provided me with new code it made up based on the file name...
Yes, I had to set the Ollama context size to 32K
Thank you, it's working as expected now!
Re: Qwen3-Coder-Next
#379Earlier quoted context omitted.
Nice! Getting ~39 tok/s @ ~60% GPU util. (~170W out of 303W per nvtop). System info: $ ./llama-server --version ggml_vulkan: Found 1 Vulkan devices: ggml_vulkan: 0 = Radeon RX 7900 XTX (RADV NAVI31) (radv) | uma: 0 | fp16: 1 | bf16: 0 | warp size: 64 | shared memory: 65536 | int dot: 1 | matrix cores: KHR_coopmat version: 7897 (3dd95914d) built with GNU 11.4.0 for Linux x86_64 llama.cpp command-line: $ ./llama-server…
Super cool! Also with `--fit on` you don't need `--ctx-size 32768` technically anymore - llama-server will auto determine the max context size!
Nifty, thanks for the heads-up!
Re: Qwen3-Coder-Next
#380Earlier quoted context omitted.
Thanks! Oh Qwen3's own GGUFs also works, but ours are dynamically quantized and calibrated with a reasonably large diverse dataset, whilst Qwen's ones are not - see https://unsloth.ai/docs/basics/unsloth-dynamic-2.0-ggufs
I've read that page before and although it all certainly sounds very impressive, I'm not an AI researcher. What's the actual goal of dynamic quantization? Does it make the model more accurate? Faster? Smaller?
More accurate and smaller.
quantization = process to make the model smaller (lossy)
dynamic = being smarter about the information loss, so less information is lost