No opus 4.7 , gpt5.5 , Gemini flash 3.5 in benchmarks
TBF, Flash 3.5 was released 2 days ago
Qwen3.7-Max: The Agent Frontier
301–310 of 317 posts
Re: Qwen3.7-Max: The Agent Frontier
#302Earlier quoted context omitted.
There are two flavors of Qwen 3.6: - A 27B "dense" model - A 35B "Mixture of Experts" model, which activates only 3B parameters for each token. For your hardware, I strongly recommend `unsloth/Qwen3.6-35B-A3B-GGUF:Q4_K_M`. I have an M1 Max with 32GB VRAM from 2021 that can read at ~300-500 tokens/sec and write at ~30 tokens/sec with llama-cpp's default settings, which is plenty fast. The 27B model can read ~70tok/sec…
For coding tasks 27B is reported to be much more effective, altho you can probably only run 4b or 5b quants @ this memory. Recommend https://www.reddit.com/r/LocalLLaMA/ as a great source for this type of discussion.
I played around with local LLMs on my M4 Max 64GB this weekend and this is exactly what I found. I put Opus 4.7 "head to head" on the same task as Qwen 3.6 and a few other local models. The 35B did not perform well IME - it needed a lot of handholding and even then the final result did not work until a few more tweaks, while Claude one shot the task. The 27B was much better and also one shot the task, but took about ~55min as opposed to about ~15min for Claude. The 27B is probably something that I could happily run for many use cases if I had some faster hardware... the main problem there seems to be that at larger context sizes, prompt decoding can take several minutes.
Re: Qwen3.7-Max: The Agent Frontier
#303Earlier quoted context omitted.
Its very hard to be so naive.
I think you are being ridiculous. Tampering with an LLMs pretraining is a difficult undertaking. There is plenty of evidence that training a model to walk the party line leaves it less capable than if it weren't. It's not very subtle manipulation either; ask qwen of Taiwan is a part of China in German and in English and only the English answer will be party-approved.
Tampering with pretraining is as simple as uploading a few pages to github. See pliny
Re: Qwen3.7-Max: The Agent Frontier
#304Earlier quoted context omitted.
The 27B model is dense, so is relatively slow. The 35B-A3B model is marginally weaker but being MoE is much faster - like ~4-8x faster in basic benchmarks on my M1 Max. For comparison, I just ran a couple of quick benchmarks (default settings) with llama-bench: Qwen3.6-35B-A3B at Q6_K_XL gave 858 t/s pp512 (prompt processing) and 43 t/s tg128 (token generation). Qwen3.6-27B at Q4_K_XL gave 103 t/s pp512 and 8 t/s tg1…
Have you tried enabling MTP? Those numbers are similar to what I was getting on my Strix Halo box, but configuring/enabling MTP doubled the TG speed of the 27B model (18-20 t/s now).
Thanks - I’m in the process. I’ve tried briefly, but so far it appears marginally slower. (Noting that llama-bench doesn’t support MTP yet so you’re reduced to running different prompts and eyeballing the log.)
So I’m assuming I’ve done something wrong along the way, but I’ve not had time yet to explore it.
Re: Qwen3.7-Max: The Agent Frontier
#305Earlier quoted context omitted.
The big question for me having used a lot of these SOTA chinese models is: what is its token efficiency like? Running Step 3.5 Flash locally for example, it's an amazingly capable model all things considered, but it's token efficiency is so bad that it gets out performed by most others wall-clock time (even with my MTP-support for it hacked in to llama.cpp: despite being trained on three heads, MTP 2 is the sweet spo…
Yes it's a big thing that people are slowly becoming more aware of. Nvidia models are even worse than Qwen! https://sql-benchmark.nicklothian.com/#token-efficiency-and-... (mouse over the cells for token counts and click for traces) Gemma 4 is good for this, as AA notes: > Gemma 4 31B is notably token efficient, using 39M output tokens to run the Intelligence Index vs 98M for Qwen3.5 27B (Reasoning). This is ~2.5x fe…
Right and all of my own evals back this up for Gemma 4...
...except its notably worse at coding in an agent context even with a harness setup to do exactly what Google says it should do (wrt. to sending summarised thinking back and so on)
So despite it being far better token efficiency wise, it's just worse for what I need to use it for compared to DSv4 Flash or Qwen 3.6 27B
Such a shame, too.
Re: Qwen3.7-Max: The Agent Frontier
#306Earlier quoted context omitted.
Can you elaborate more on the differences in running ollama or lmstudio? Do they actually slow down the speed of the inference and if so why? Or is it just a preference thing?
Ollama and LM-Studio are fine. Their main advantage is that they have a nice way to browse models -- LMStudio from huggingface and Ollama from their own curated list. Both are great ways of getting started. Pick LM-Studio if you'd like a nice GUI frontend to mlx-lm or llama-cpp; pick ollama if you'd like a nice command line interface and don't need non-default parameters. LM-Studio doesn't support certain parameter c…
Thank you, that makes a lot of sense
Re: Qwen3.7-Max: The Agent Frontier
#307Earlier quoted context omitted.
Compared to what we have proof the US government have engaged in before? Do people not remember PRISM anymore? It was virtually impossible to think of the scope before it was leaked, and you'd be marked as a conspiracy theorist for believing that happened, before it was made concretely true. I think it's borderline naive to assume various agencies haven't infiltrated OpenAI, Anthropic and others, essentially the enti…
Agencies like the CIA have infiltrated the news agencies, so they have indirect power over the information that LLMs consume.
Yup, that's data poisoning on worldwide scale.
Re: Qwen3.7-Max: The Agent Frontier
#308Earlier quoted context omitted.
Don't mind me asking, but where did you find $5k RTX 6000? Even 48GB model (previous gen) shows minimum at 7k, and 96GB one (Blackwell) is ~10k on Amazon...
$5K is presumably what it costs to pay some local gangsters to break into an nVidia warehouse. That's the only you will pay $5K for an RTX 6000 for the next couple of years. The server edition has gone up $2K in the last couple of weeks alone, at the outlet where I bought one previously.
Man... Does it mean that buying RTX 6000 for 10k today is actually becoming an investment?
Re: Qwen3.7-Max: The Agent Frontier
#309Earlier quoted context omitted.
I had a buddy in high school that was notorious for doing the same thing. (He's now a senior director at a Big 4 consultancy. :) )
Do you mind expanding a little more?
They had a buddy who used to lie a lot when they were younger… now they get paid for it
Re: Qwen3.7-Max: The Agent Frontier
#310Earlier quoted context omitted.
If your hardware fits K_M but not K_XL, should you prefer going down to a lower quantization’s XL or sticking to the higher quant’s Q_M?
The correct answer should be "try it!" But as models are starting to pack more information into less bits, some weights are just going to end up becoming super important and very sensitive to quant. So, I'd just move down a Q size, and continue with K_XL. Like, I'm betting Q3_K_XL will beat Q4_K_M on any given model in real world testing, even though its ~20% smaller, but perform worse on benchmaxxing. The only excep…
[dead]