I was getting dangerously close to my weekly Claude Code limit last night so I had Claude set up Qwen3.6 with llama.cpp and OpenCode. Honestly it's a great (free!) alternative to Claude Code--certainly more than good enough for a lot of smaller less complex tasks. I'm excited to try this new version. The fact that open-source models are so close to the frontier is very impressive.
This one doesnt seem to be open source though sadly. Using chinese servers is a step to far for me personally
Qwen3.7-Max: The Agent Frontier
281–290 of 317 posts
Re: Qwen3.7-Max: The Agent Frontier
#282Earlier quoted context omitted.
My computer lacks the ram.
Well if it had all the ram, you still could not run it because you can't download it. It's hosted by them, just like chatgpt or claude.
BTW - They’re still censoring human rights violations.
Re: Qwen3.7-Max: The Agent Frontier
#283Earlier quoted context omitted.
Aider is still around? That is pre-tool-calling era stuff. Better compare against Pi.
I just started running coding agents locally. So you recommend Pi over opencode? (And obviously aider is out?)
Re: Qwen3.7-Max: The Agent Frontier
#284Earlier quoted context omitted.
how could running the qwen GGUF phone home? that would require cooperation with the inference backend (llama-cpp), or some kind of model exploit. It’d be far easier to pay the agent harness devs or supply-chain some plugin or something, that space is the Wild West anyways I've certainly used these models without wifi without any differences.
You've used Qwen with model quantization, locally without internet connection. A lot of people are purchasing access via Alibaba Cloud directly, or indirectly by companies which host the model.
Re: Qwen3.7-Max: The Agent Frontier
#285Earlier quoted context omitted.
There are two flavors of Qwen 3.6: - A 27B "dense" model - A 35B "Mixture of Experts" model, which activates only 3B parameters for each token. For your hardware, I strongly recommend `unsloth/Qwen3.6-35B-A3B-GGUF:Q4_K_M`. I have an M1 Max with 32GB VRAM from 2021 that can read at ~300-500 tokens/sec and write at ~30 tokens/sec with llama-cpp's default settings, which is plenty fast. The 27B model can read ~70tok/sec…
MTP recommended
Re: Qwen3.7-Max: The Agent Frontier
#286Earlier quoted context omitted.
May I ask why the M instead of XL? Obviously bigger != better but I don't know what the differences are.
These are dynamic quants, and they're basically just an indication of how far away from the desired quant it is allowed to go to achieve the goal. Generally, unsloth's toolchain moves quants up, rarely down. * _0 and _1 do not use K quant and scales 32x32 blocks according to the original (B)F16 values; _0 scales the block using the original max and min values. _1 does this per row instead of per block. * K quants do…
Re: Qwen3.7-Max: The Agent Frontier
#287Earlier quoted context omitted.
These are dynamic quants, and they're basically just an indication of how far away from the desired quant it is allowed to go to achieve the goal. Generally, unsloth's toolchain moves quants up, rarely down. * _0 and _1 do not use K quant and scales 32x32 blocks according to the original (B)F16 values; _0 scales the block using the original max and min values. _1 does this per row instead of per block. * K quants do…
You seem to understand this stuff pretty well, any recommendations on resources (blogs, YouTube channels, whatever) for software engineers that want to keep up with this stuff on this kind of level? A lot of the content about AI out there is kind of produced to the lowest common denominator. Basically a never ending scheme of get rich quick/passive income kinds of AI content.
If you’re curious about what a particular switch does, clone the llama-cpp repository to your computer and try asking your favorite pet rock prompts like “This is llama-cpp. Can you look at what the -ctk parameter does and explain to me?” Giving Claude/codex/whatever access to the actual code goes a long way, but it is just one opinion.
If you’d like to learn how transformer-based language modeling works in detail, I suggest starting with chapter 0 or 1 of https://arena-chapter0-fundamentals.streamlit.app/ depending on your skill level, then use that to work your way to reading research papers.
Graduate students who study these topics are generally as annoyed by the “get rich quick” style of advertising as you are, so the deeper you go toward academic research the quieter those voices tend to get, mercifully. That said, this is balanced by the unfortunate fact that top labs have strong posturing signals they try to send, so it can be hard to see which preprints actually have good ideas, which are trying to promote their group’s tech instead of doing science out of curiosity, and which have authors who’ve innocently deluded themselves into overfitting their own pet projects. Read widely but adversarially, test everything but hold fast to the good stuff, etc etc
Re: Qwen3.7-Max: The Agent Frontier
#288Earlier quoted context omitted.
There are two flavors of Qwen 3.6: - A 27B "dense" model - A 35B "Mixture of Experts" model, which activates only 3B parameters for each token. For your hardware, I strongly recommend `unsloth/Qwen3.6-35B-A3B-GGUF:Q4_K_M`. I have an M1 Max with 32GB VRAM from 2021 that can read at ~300-500 tokens/sec and write at ~30 tokens/sec with llama-cpp's default settings, which is plenty fast. The 27B model can read ~70tok/sec…
Using omlx on the M1 max I get about 15tps from 27b
Re: Qwen3.7-Max: The Agent Frontier
#289Earlier quoted context omitted.
And for what? Spend 10-15k for the slopiest of slop code, non deterministic automations, and the ability to spawn an AI gf? This whole thing is really starting to remind me of the crypto hype phases of 2016-2018 when everyone thought their investment in GPUs was going to make them rich.
[dead]
Re: Qwen3.7-Max: The Agent Frontier
#290Earlier quoted context omitted.
These are dynamic quants, and they're basically just an indication of how far away from the desired quant it is allowed to go to achieve the goal. Generally, unsloth's toolchain moves quants up, rarely down. * _0 and _1 do not use K quant and scales 32x32 blocks according to the original (B)F16 values; _0 scales the block using the original max and min values. _1 does this per row instead of per block. * K quants do…
If your hardware fits K_M but not K_XL, should you prefer going down to a lower quantization’s XL or sticking to the higher quant’s Q_M?
But as models are starting to pack more information into less bits, some weights are just going to end up becoming super important and very sensitive to quant. So, I'd just move down a Q size, and continue with K_XL. Like, I'm betting Q3_K_XL will beat Q4_K_M on any given model in real world testing, even though its ~20% smaller, but perform worse on benchmaxxing.
The only exception I could think of is quantizing small models, like, my testing on Gemma E2B/E4B and Qwen 3.5 9B, quantizing at all was super noticeable... they can't spread the error across more weights.
Good news (at least for me), 24GB of VRAM is enough to store either of those in BF16 and then a ton of room for F16/F16 KV cache.