Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Opus 5 is currently #1 on Artificial Analysis Intelligence Leaderboard

artificialanalysis.ai

Post reply on HN
Print view View original

171–180 of 251 posts

Re: Opus 5 is currently #1 on Artificial Analysis Intelligence Leaderboard

#171

Post by alex43578 »

andriy_koval wrote: post #87
mchusma wrote: post #42

Earlier quoted context omitted.

Gemini 3.1 pro is really good for knowledge tasks. Google has done well there. And image analysis with Gemini flash 3.6 is solid. It’s just anything coding or agentic they fall short.

they likely tune their models for areas where they have their money: search, ads, youtube, etc.

I wonder if that'll be a mistake as LLMs are used for internal LLM R&D. Either Google will not take this approach, use a 3rd party model (weird, data leak risk?), or use a non-public internal model (big sunk dev cost with no recoup by trickling it to public).

Re: Opus 5 is currently #1 on Artificial Analysis Intelligence Leaderboard

#173

Post by Gareth321 »

xrisk wrote: post #158
zwaps wrote: post #153

Earlier quoted context omitted.

Sol is a complete mess for me. It only works on end to end tasks in fresh codebases. Otherwise it cannot follow instructions, changes and deletes unrelated features or does sloppy work to mark a task completed while leaving a compromised codebase. I could not get Sol to finish a feature in a complex code base without several loops of fixing and reverting

Might be an indication that your task unit is too unstructured or your code base is a mess. Is your actual code doing something complex or is this incidental complexity? Relying on the model’s “intelligence” to patch over these issues hasn’t proven to be a reliable strategy for me. Of course, this might not apply to you, just my 2 paisa.

A well structured architecture, repo, and requirements document with concrete small deliverables can be competently delivered with cheap Chinese models. We rely on frontier models so that we don't have to spend several days/weeks on planning/architecture/documentation. That's their value proposition: superior intelligence. If they can't deliver that, they're useless at the current price.

Re: Opus 5 is currently #1 on Artificial Analysis Intelligence Leaderboard

#174

Post by Gareth321 »

zwaps wrote: post #153
theplumber wrote: post #79

Earlier quoted context omitted.

Opus5 is simply not as inteligent as Sol max. To me it looks worse than Opus 4.8 on some tasks. When I say worse I mean mainly superficial. I basically have to teach him how the whole app/framework works before he just jumps doing stupid stuff(I.e adding features already supported but in a different form)

Sol is a complete mess for me. It only works on end to end tasks in fresh codebases. Otherwise it cannot follow instructions, changes and deletes unrelated features or does sloppy work to mark a task completed while leaving a compromised codebase. I could not get Sol to finish a feature in a complex code base without several loops of fixing and reverting

I agree. I tried hard to use Sol to its full potential but it makes sloppy mistakes. I suspect a part of this is the newly reduced context limit. Performance gets worse the more memory compactions have occurred. But I also subjectively notice a micro-focus temperament. Developing and deploying small parts of the project to a high degree while forgetting how that component fits into the larger architecture. Once it's complete it realises that what it built doesn't align well with everything else, then it rewrites the component (and other components). Rinse and repeat. I have to be much more prescriptive with Sol, which makes it much less useful for me. If I have to be prescriptive, I can use the Chinese models and achieve the same thing.

Opus provides genuine insights. Things I have not considered and have missed. More importantly, it allows me to skip most of the architecture decisions and requirements work. That's a genuine time saver for me.

Re: Opus 5 is currently #1 on Artificial Analysis Intelligence Leaderboard

#175

Post by krzyk »

firasd wrote: post #14

Very interesting that one of the components is "AA-Omniscience Index" AA-Omniscience Index (higher is better) measures knowledge reliability and hallucination. It rewards correct answers, penalizes hallucinations, and has no penalty for refusing to answer. This seems to be a good proxy for param size/density and the ranking breaks down as such: Claude Fable 5 (with fallback), Gemini 3.1 Pro Preview, Claude Opus 5 (Ma…

If it doesn't give penalty for refusal it is not that useful.

Re: Opus 5 is currently #1 on Artificial Analysis Intelligence Leaderboard

#176

Post by meander_water »

The funny thing is that these leaderboards have become completely meaningless for end-users to make decisions on when to use what model.

A single metric ranking is useless because each model has strengths and weaknesses for specific domains and tasks. There is no "one best model" anymore, and you might not even need the best model for the level of complexity for your task.

For e.g. you might use Fable for UI design, Sol for systems design backend work and Kimi K3 for exploit development.

The only purpose these metrics serve is bragging rights for the model companies.

Re: Opus 5 is currently #1 on Artificial Analysis Intelligence Leaderboard

#177

Post by krzyk »

weird-eye-issue wrote: post #160
vikramkr wrote: post #66

Earlier quoted context omitted.

I doubt there's any sort of criminal behavior there - the model is anthropic's up and anthropic probably charges a very expensive license fee that's the same for all of them, and their cogs on compute aren't going to be wildly different, so the main drivers of the cost are roughly the same and they're all offering customers the same end product so the prices would likely also be similar in the end

Requiring the exact same product to be set at a specific price across providers would not be criminal behavior, lol

It is breaking competition.

Would you like all products everywhere be priced like their producers want?

Re: Opus 5 is currently #1 on Artificial Analysis Intelligence Leaderboard

#178

Post by krzyk »

kccqzy wrote: post #50
gck1 wrote: post #41

Earlier quoted context omitted.

Every time an online chatter (e.g. "limits are better", "model is better") makes me to reevaluate my principle of never paying Anthropic, I go to the model card, which strengthens my belief in the principle. Why is Anthropic is so hell-bent on this auto/silent downgrade? Do they have a single user who prefers an auto-lobotomization instead of a refusal? Have they learned nothing from the backlash the first time?

Just go to /config. The very second configuration item is “Switch models when a message is flagged” and presumably you want to turn this off. Oh but then you said you never pay Anthropic so you haven’t actually used Claude Code yet. Why would anyone listen to the opinion of a non-user?

So when you turn that off will you get answer from Fable?

Re: Opus 5 is currently #1 on Artificial Analysis Intelligence Leaderboard

#179

Post by krzyk »

ricardobeat wrote: post #33
chmod775 wrote: post #20

The more interesting finding is that it's still the second most expensive model (after Fable 5) by a long shot. At least two models (GPT-5.6, Kimi K3) match its score (~1-2% diff) for half the cost.

The chart shows max effort, used mostly by price-insensitive enterprise users. At medium effort it drops to almost half K3’s cost, and is probably sufficient for 95% of coding tasks.

Enterprise users are price sensitive, because they are charged per token (and have limit per user set by companies, and those limits are different from $50 per month to 1500$ per month). Subscription users might be insensitive to that.

Re: Opus 5 is currently #1 on Artificial Analysis Intelligence Leaderboard

#180

Post by king_phil »

theplumber wrote: post #46

Why do I find it dummer/even more superficial than opus 4.8? It just continued a session and I had to stop it because it become obviously “lost”

Massive performance degradation is expected when continuing a session with a different model
Post reply on HN