Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Qwen3.8 Max now ranked as the best overall model by agentic index

artificialanalysis.ai

Post reply on HN
Print view View original

51–60 of 364 posts

Re: Qwen3.8 Max now ranked as the best overall model by agentic index

#51

Post by DC-3 »

quotemstr wrote: post #47
petercooper wrote: post #40

Hopefully this boils down to the smaller versions they've teased. In my experience, Qwen models are the closest to the "less knowledge, more intelligence" (yes, the two are hugely correlated!) ideal some tool-dependent tasks need. Even the 3.5 2B can be easily prompted to always lean on tools and not jump to false conclusions (although its actual coding skills are abysmal, as you'd expect).

> less knowledge, more intelligence People produce such models by over-RL-ing smaller models on math and coding tasks. I've found the results capable of neither innovative work nor thinking outside the box. They're straight-A students raised by tiger moments who never let them play freely for hours in the dirt. Perhaps you could say such models are skilled --- but intelligent? Not by my measure. People and AIs alike…

It's amusing to me that AI has become sophisticated enough that people have started being racist to it.

Re: Qwen3.8 Max now ranked as the best overall model by agentic index

#52

Post by thefourthchime »

eli wrote: post #5

I believe it. It's extremely good at troubleshooting. I gave Qwen and Kimi K3 the same annoying, complicated, intermittent bug to track down. Kimi did a bit better in understanding the existing code, but Qwen built some diagnostic tools and did an excellent statistical analysis on the log data. Qwen got way closer to the truth. I'm very much looking forward to their forthcoming smaller model Qwen 3.8 releases. A vers…

Did you also try Opus 5 and 5.6 Sol?

Re: Qwen3.8 Max now ranked as the best overall model by agentic index

#53

Post by LoganDark »

syntaxing wrote: post #3

I am so excited for Qwen 3.8 27B. It’s a shame how slow prefill (~3-400) is on a strix halo but it’s such a good model for agentic tasks.

I find that 35B-A3B is much easier to run on my M4 Max (both prefill and generation)

Re: Qwen3.8 Max now ranked as the best overall model by agentic index

#54

Post by criley2 »

drnick1 wrote: post #4

Why does an open weights model cost nearly the same as GPT5.6? $1.14 vs $1.23 on the cost index. Since you can't presumably run this on your own hardware given the model size and hence gain other things like privacy, I don't see any reason to move away from GPT at this rate.

GPT5.6Sol completes the suite in 70M tokens, while Qwen3.8Max needs like 145M tokens. So this is a case where models like Qwen 3.8 and Kimi K3 use a lot more output (reasoning) tokens, go a good bit slower, so they can ultimately achieve a better intelligence score than if they went more quickly.

There are a couple of frontiers (ok bad word, maybe categories) in open weight models.

These Qwen 3.8 and Kimi K3 style models aren't trying to win on price, they're trying to compete on intelligence and capability.

Models like Deepseek V4 Flash (updated this week) are $0.03 a task, or 50X cheaper than Qwen3.8/Kimi K3, and 100X cheaper than Fable, while offering stunning intelligence. That's a different frontier for competition, and perhaps one more interesting for someone who wants to see them compete on cost.

Re: Qwen3.8 Max now ranked as the best overall model by agentic index

#56

Post by superfrank »

SwellJoe wrote: post #8

I find that surprising. I've been trying it on several projects and have found it's pretty sloppy. It leaves stuff broken, doesn't reliably write tests to check its own work unless explicitly prompted, misunderstands the assignment, etc. It is smart and reasonably quick but not reliable.

I've come to the same conclusion over and over with all of the Chinese models that have been claimed to be catching up with OpenAI's and Anthropic's frontier models (Deepseek 4, GLM 5.2, Kimi K3).

At their best, I think they're closing in on Opus and GPT, but they're incredibly inconsistent and the variance in output quality is much higher than the best from any of the Anthropic or OpenAI models from the last few generations. The only way I can describe it is that it feels like a lack of intuition with the models which means I find my self needing to write longer prompts or have more back and forth to get them to do what I want from them.

To give an example, I have a saved prompt that I use as a sanity check on some data I'm storing. It reads about 50 rows from a DB and matches them to the UI and makes sure the data is displaying correctly. I've been using this with GPT 5.5 and now 5.6 for a few months and running it a few times a week with no issue. Sometimes I'll run it multiple times in a single chat if I notice bad data (run it, fix thing, run again, fix another thing).

I recently tried to switch to using Deepseek v4 (first flash and then pro) and while both did the task just fine, both would do things like change the response format from one message to another in the same chat or randomly decide to omit things it didn't think were relevant. At one point I ran the prompt, fixed some bad data, and then said "Okay, I fixed row 7, run {prompt} again" and so it decided to leave row 7 out of the response. A few times the first message would contain a table and then the next run in the same chat would contain the data in a bulleted list.

None of those are major issues and all could be solved with a bit more rigor in my prompting, but for me it makes them harder to work with. Those examples are a bit trivial, I think they're the easiest way for me to illustrate the gaps I see with them.

Re: Qwen3.8 Max now ranked as the best overall model by agentic index

#57

Post by Art9681 »

eli wrote: post #6
drnick1 wrote: post #4

Why does an open weights model cost nearly the same as GPT5.6? $1.14 vs $1.23 on the cost index. Since you can't presumably run this on your own hardware given the model size and hence gain other things like privacy, I don't see any reason to move away from GPT at this rate.

It's not enough that it's better? Many providers will host it and will compete on price. It also can't easily be taken away because one company (or one government) decides they don't want it around any more. People can fine-tune it for particular workloads.

They cherrypicked benchmarks. The ONE weighed benchmark where is beats Opus5 by 0.1 points is what was linked because that's how propaganda works. The Agentic Index that includes the full benchmark suite has it in 5th place.

Might as well use gpt-sol.

Re: Qwen3.8 Max now ranked as the best overall model by agentic index

#58

Post by ben8bit »

Haven't tried this yet, but going to soon! I have to wonder what happened at Anthropic. We've cancelled our subscription in favor of OpenCode & Codex. Sol is just so good & OC goes so far for every $ spent. Claude's become a pain to work with - average output with an annoying personality. Who knew this would be an issue even a year ago? In any case, loving the stuff from the Chinese models!

Re: Qwen3.8 Max now ranked as the best overall model by agentic index

#59

Post by Art9681 »

quirino wrote: post #26

A couple days ago they had published an overall score of 53 for this model, but that was removed and today it returned with a score of 56. I wasn't able to find an explanation from them. Anyone knows what happened?

A wire transfer happened.

Re: Qwen3.8 Max now ranked as the best overall model by agentic index

#60

Post by theropost »

Anthropic is a bit nuts, I had $260 of credits on my max account for the extra usage the other night. It was expiring, so I figured I'll fire up an agentic swarm to deep dive and make some deep changes to some old cold bases.. literally 25 minutes or less, $260 burnt, it didn't get get into the implementation, just wrote a ton of useless plans for the most part. It really opened my eyes to what they expect to charge people.. wayyyy overpriced.
Post reply on HN