Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Qwen3.8 Max now ranked as the best overall model by agentic index

artificialanalysis.ai

Post reply on HN
Print view View original

211–220 of 364 posts

Re: Qwen3.8 Max now ranked as the best overall model by agentic index

#211

Post by hungryhobbit »

seizethecheese wrote: post #97

Opus is still first in Intelligence Index followed by Fable, GPT 5.6, Kimi K3 then Qwen 3.8 max. https://artificialanalysis.ai/#intelligence Our leaderboard combines Arena ELO, AA Intelligence index, latency and speed and goes: #1 Opus 5 #2 Kimi K3 #3 Qwen3.8 Max #4 GPT 5.6 Sol Source: http://pellmell.ai/leaderboard . This jumps around a lot based on the top throughput and latency of whatever provider happens to be b…

It seems like if latency is having such a big effect that it's changing the winners, maybe your tests are awful and shouldn't be so latency dependent?

I mean, I get it: how fast a model responds is relevant. But a test that changes second by second is far less relevant than a test that tells you how smart the model is, and accounts for latency in some way that isn't constantly changing the result.

Re: Qwen3.8 Max now ranked as the best overall model by agentic index

#212

Post by zmmmmm »

The fact that the Chinese models have caught up on benchmarks suggests to me that its likely we will start to transition now into much more of a brand war. It will be subjective qualities that drive our decisions more than measures of absolute intelligence. Already I am choosing models more because I like the personality or style of what they do than because I think they have the absolute highest chance of outputting the most technically correct answer to any given prompt. It will be very interesting to see how things evolve in this direction.

Re: Qwen3.8 Max now ranked as the best overall model by agentic index

#213

Post by veber-alex »

veber-alex wrote: post #204
d4rkp4ttern wrote: post #116

Earlier quoted context omitted.

All these "intelligence" benchmarks miss something extremely important when using an LLM in a code-agent harness: How it communicates with you about what it did. Opus-5 is practically unusable (for complex tasks) in this sense - its updates are voluminous, and dense with cryptic language (there are numerous reddit threads complaining about this, so it's not just me). I often have to ask it to re-state concisely in pl…

It takes 2 minutes to fix Opus 5 https://code.claude.com/docs/en/output-styles

lol. I am being downvoted for trying to help people out.

This community is pure trash.

Re: Qwen3.8 Max now ranked as the best overall model by agentic index

#214

Post by dgellow »

nomel wrote: post #76
onomojo wrote: post #34

Any benchmark showing Opus 5 as the best just loses credibility for me. Anyone who's actually used Opus 5 daily knows what I'm talking about.

What's the clear best, that you see?

Hilarious to see only different responses

Re: Qwen3.8 Max now ranked as the best overall model by agentic index

#215

Post by michelsedgh »

AustinDev wrote: post #210
doginasuit wrote: post #198

Earlier quoted context omitted.

Another potential takeaway is that the models all gathering around the same point supports the idea that there is a ceiling to LLM capability.

They always all gather around the same spot then that spot moves every 6-9 months. I think the clustering is more likely evidence of distillation. I don't personally think distillation is a bad thing. If the LLM providers can distill all of human output into their models for 'free'. I don't think distilling a model from the output of those models is morally wrong.

What an interesting take. One question, do you think stealing from a thief is morally okay? I'm just asking no judgement on my side.

Re: Qwen3.8 Max now ranked as the best overall model by agentic index

#216

Post by bitexploder »

Zambyte wrote: post #186
jjcm wrote: post #166

China has caught up is the main takeaway here. The SOTA models are so close that it's really hard to compare them intelligence wise - you have to get a feel for them yourself and what works for you. What I'm really excited for is the 27B model. 3.6 is still the king of local, and if 3.8 makes the same improvements it could really legitimately make local viable as a default. I'd love to run a perpetual agent on 3.8 th…

Qwen 3.6 27b is already a viable default. I'm running it on a single 7900 XTX right now for Go development with pi. It's great.

I find 35B A3B viable as well, but your harness and runtime really matters to get tool calling and such dialed in. In fact, I would encourage you to experiment with it some as I find I get more reliable output from 35B A3B, though 27B is still generally smarter. A3B with a review cycle or two from 27B is great for me.

One of the reasons is, with good specs and design, A3B is just so fast. It isn't as smart as the 27B model, but it is close enough it can usually figure it out with the right tools.

Re: Qwen3.8 Max now ranked as the best overall model by agentic index

#217

Post by seizethecheese »

hungryhobbit wrote: post #211
seizethecheese wrote: post #97

Opus is still first in Intelligence Index followed by Fable, GPT 5.6, Kimi K3 then Qwen 3.8 max. https://artificialanalysis.ai/#intelligence Our leaderboard combines Arena ELO, AA Intelligence index, latency and speed and goes: #1 Opus 5 #2 Kimi K3 #3 Qwen3.8 Max #4 GPT 5.6 Sol Source: http://pellmell.ai/leaderboard . This jumps around a lot based on the top throughput and latency of whatever provider happens to be b…

It seems like if latency is having such a big effect that it's changing the winners, maybe your tests are awful and shouldn't be so latency dependent? I mean, I get it: how fast a model responds is relevant. But a test that changes second by second is far less relevant than a test that tells you how smart the model is, and accounts for latency in some way that isn't constantly changing the result.

Latency isn't changing the results for the coding index or arena ELO, but neither of those take latency or throughput into account, so we added those to our leaderboard as score components.

Latency and throughput matter a ton as a user, so I think it's actually totally defensible for a leaderboard to bounce around a lot as these numbers change. The best model to use changes a lot based on these!

Re: Qwen3.8 Max now ranked as the best overall model by agentic index

#218

Post by combyn8tor »

onomojo wrote: post #34

Any benchmark showing Opus 5 as the best just loses credibility for me. Anyone who's actually used Opus 5 daily knows what I'm talking about.

It works fine for me. Only issue I have is that it has me constantly reaching for the dictionary.

Re: Qwen3.8 Max now ranked as the best overall model by agentic index

#219

Post by bitexploder »

comboy wrote: post #30
eli wrote: post #5

I believe it. It's extremely good at troubleshooting. I gave Qwen and Kimi K3 the same annoying, complicated, intermittent bug to track down. Kimi did a bit better in understanding the existing code, but Qwen built some diagnostic tools and did an excellent statistical analysis on the log data. Qwen got way closer to the truth. I'm very much looking forward to their forthcoming smaller model Qwen 3.8 releases. A vers…

How CLI are you guys using for qwen and kimi?

OpenRouter and Pi here now.

Re: Qwen3.8 Max now ranked as the best overall model by agentic index

#220

Post by AussieWog93 »

michelsedgh wrote: post #215
AustinDev wrote: post #210

Earlier quoted context omitted.

They always all gather around the same spot then that spot moves every 6-9 months. I think the clustering is more likely evidence of distillation. I don't personally think distillation is a bad thing. If the LLM providers can distill all of human output into their models for 'free'. I don't think distilling a model from the output of those models is morally wrong.

What an interesting take. One question, do you think stealing from a thief is morally okay? I'm just asking no judgement on my side.

I'd say it's more "Downloading LimeWire Pro from LimeWire" than actual theft.
Post reply on HN