Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Opus 5 is currently #1 on Artificial Analysis Intelligence Leaderboard

artificialanalysis.ai

Post reply on HN
Print view View original

231–240 of 251 posts

Re: Opus 5 is currently #1 on Artificial Analysis Intelligence Leaderboard

#231

Post by rixed »

ffsm8 wrote: post #159
colinhb wrote: post #157

Earlier quoted context omitted.

Opus 5 hasn't been available for that long - long enough for benchmarks, but not really use and develop a subjective view on

I suspect most of those comments on llms like the parents are generated by anthropic and openai to shape the discussion/mindset They always give off the same astroturfing vibes that reddit became infested with after the early 2010s (just look at it's comment history) Ofc unprovable for users. Ycombinatior could try to, but it'd just become a cat/mouse game which they'd likely lose because of the incentives

That's not necessary. Sometimes we make choices and we feel the need to justify them in vivid flamewars regardless of how arbitrary. Vi vs emacs or amd vs intel, now anthropic vs openai.

We love to take sides, to belong to a small community of peers, no?

Re: Opus 5 is currently #1 on Artificial Analysis Intelligence Leaderboard

#233

Post by fireant »

aarondong wrote: post #2

Before getting too excited, take a look at the intelligence vs cost matrix: https://artificialanalysis.ai/models?intelligence-index-toke...

That index really needs harder tasks so that it's not just a benchmark of what model is cheapest

Re: Opus 5 is currently #1 on Artificial Analysis Intelligence Leaderboard

#234

Post by zwaps »

theplumber wrote: post #162
zwaps wrote: post #153

Earlier quoted context omitted.

Sol is a complete mess for me. It only works on end to end tasks in fresh codebases. Otherwise it cannot follow instructions, changes and deletes unrelated features or does sloppy work to mark a task completed while leaving a compromised codebase. I could not get Sol to finish a feature in a complex code base without several loops of fixing and reverting

It happened to me as well but in a different direction: i.e adds non library code in a shared library. Another issue I with GPT is that it is chasing too much edge cases/security issues(I.e chasing ghosts). However this makes it also a strong model because it fixes/solves problems that both Opus and Fable are incapable. In reviews it catches bugs that both Fable and Opus are missing to spot. To me the “best of both w…

But then Sol doesn’t really follow the plan or rather invents things it should add or cut on top of the plan. How does one deal with that?

Re: Opus 5 is currently #1 on Artificial Analysis Intelligence Leaderboard

#235

Post by charcircuit »

vikramkr wrote: post #225
krzyk wrote: post #177

Earlier quoted context omitted.

It is breaking competition. Would you like all products everywhere be priced like their producers want?

The competition is between openai and anthropic, if there are price agreements between them that's absolutely price fixing. Or if there's collusion between the cloud providers to inflate compute. I would expect Amazon and gcp to both pay about the same in license fees to anthropic for their models though because they're paying for the same thing. If I buy an apple for a dollar at one store and an apple for a dollar a…

>The competition is between openai and anthropic

In capitalism there are many many competitions going on at the same time. Both models can compete and inference providers can compete for costs.

In your apple example if at the same stores you saw "open" apples having different prices and having sales you might question if the costs of those frontier apples are not being manipulated.

Re: Opus 5 is currently #1 on Artificial Analysis Intelligence Leaderboard

#236

Post by charcircuit »

weird-eye-issue wrote: post #183
krzyk wrote: post #177

Earlier quoted context omitted.

It is breaking competition. Would you like all products everywhere be priced like their producers want?

You seem terribly confused. Manufacturers are almost always able to set prices. That is not anti-competitive because it does not imply they are colluding with their competition...

A fixed price does not let stores compete with each other. Blocking competition is anticompetitive.

Re: Opus 5 is currently #1 on Artificial Analysis Intelligence Leaderboard

#237

Post by andai »

mejutoco wrote: post #203
andai wrote: post #186

Earlier quoted context omitted.

A year ago a new Best Model came out, and I was very excited. I tested it on a simple programming task (which required making 3 trivial changes in 3 files). The model did fine. Then I tested its little brother, the older, smaller variant of the same model. It also did fine, except it did it 3x faster and cost 9x less. In this moment, andai was enlightened.

is there a benchmark that uses prices or speed as one of the axis, in addition to accuracy? Best could mean different things to different people.

http://deepswe.datacurve.ai/ has graphs for Cost, for Token Usage, and for Agent Steps. They don't have one for Time, sadly, but Output Tokens and Agent Steps (which appear to produce near-identical rankings) are a decent proxy.

https://cognition.com/blog/frontier-code has a dropdown selector for Tokens, Cost, Time, Agent Steps, and more.

---

As a side note, these two benchmarks appear to be more sensitive at distinguishing supposedly frontier models from each other. But they themselves cannot agree on which is better!

One argues that the other has a flawed methodology (and makes a fair case). However it might also just be that the frontier is a little jagged, and sometimes one model will do better than another.

That's been my experience anyway. When I have a very important task, I always make sure to get a second opinion: I get them both to give it a shot, and then to critique each other's solutions. You can actually apply this at every stage (the review, the planning, the implementation), if you have the patience for it.

(Would be nice if there were a way to automate this process. Maybe with one of the higher level meta-harnesses that runs Claude and Codex as subprocesses. But I haven't looked into that yet...)

At any rate, combinations of models have always been found to do significantly better than a single one (e.g. see Model Alloys, Model Fusion, etc.). So make sure to use them, when appropriate!

Hope this helps.

Re: Opus 5 is currently #1 on Artificial Analysis Intelligence Leaderboard

#238

Post by braebo »

theplumber wrote: post #79
didibus wrote: post #56

What's interesting is this: The top AI models by Intelligence Index are: 1. Claude Opus 5 (Adaptive Reasoning, Max Effort) (61), 2. Claude Opus 5 (Adaptive Reasoning, Xhigh Effort) (60), 3. Claude Fable 5 (Adaptive Reasoning, Max Effort, Opus 4.8 Fallback) (60), 4. GPT-5.6 Sol (max) (59), and 5. Claude Opus 5 (Adaptive Reasoning, High Effort) (59). Which means Opus5 at Xhigh is still smarter than Sol at max, and Opus…

Opus5 is simply not as inteligent as Sol max. To me it looks worse than Opus 4.8 on some tasks. When I say worse I mean mainly superficial. I basically have to teach him how the whole app/framework works before he just jumps doing stupid stuff(I.e adding features already supported but in a different form)

My experience is the opposite. GOT 5.6 Sol is the dumbest most dangerous frontier model I’ve ever used. It actively introduces bugs and hacks and lies about what it did. Its code is almost always slop that can’t make it through even a brief review without half a dozen wtf moments. Opus is always cleaning up the terrible mess and GPT models are banned now from work because of how harmful and mind numbing stupid they are. R.e. Opus doing the wrong thing — my environment always gives the required context or leaves a paper trail so I don’t have that issue with Claude models.

Re: Opus 5 is currently #1 on Artificial Analysis Intelligence Leaderboard

#239

Post by braebo »

zwaps wrote: post #234
theplumber wrote: post #162

Earlier quoted context omitted.

It happened to me as well but in a different direction: i.e adds non library code in a shared library. Another issue I with GPT is that it is chasing too much edge cases/security issues(I.e chasing ghosts). However this makes it also a strong model because it fixes/solves problems that both Opus and Fable are incapable. In reviews it catches bugs that both Fable and Opus are missing to spot. To me the “best of both w…

But then Sol doesn’t really follow the plan or rather invents things it should add or cut on top of the plan. How does one deal with that?

I’ve spent over 100 hours trying to get GPT 5.6 Sol to resemble something useful as opposed to something actively detrimental. I concluded that it is not possible because it is fundamentally not intelligent the way Claude models are.

Re: Opus 5 is currently #1 on Artificial Analysis Intelligence Leaderboard

#240

Post by vikramkr »

charcircuit wrote: post #235
vikramkr wrote: post #225

Earlier quoted context omitted.

The competition is between openai and anthropic, if there are price agreements between them that's absolutely price fixing. Or if there's collusion between the cloud providers to inflate compute. I would expect Amazon and gcp to both pay about the same in license fees to anthropic for their models though because they're paying for the same thing. If I buy an apple for a dollar at one store and an apple for a dollar a…

>The competition is between openai and anthropic In capitalism there are many many competitions going on at the same time. Both models can compete and inference providers can compete for costs. In your apple example if at the same stores you saw "open" apples having different prices and having sales you might question if the costs of those frontier apples are not being manipulated.

Yeah there's a brand premium. Like literally with apples the ones with trademarked names can cost more. And the ones with trademarked names have an organization behind them that promote that apple variety and set fees etc for growing and selling those apples. And customers are willing to pay more because those apples usually taste way better (the group exists to stop growers from enshittifying the apple by selecting for yield over flavor like what happened to honey crisp). Those prices are being "manipulated" but that's not criminal behavior - it's not illegal and wouldn't make sense to try and make illegal. The frontier models also do have different coats and have "sales" (for personal plans, the amount of usage you can get on the 200 dollar plans is orders of magnitude more than you could get for a similar cost for any open model - you'd need the same capability at the same token efficiency at literally 1/40th the cost to be able to be cheaper) - to the extent that if there is illegal stuff going on it seems more likely to me that it's on the category of dumping/pricing unreasonably low to kill competitors in some anti-competitive way (though as I understand it it's not something courts tend to find as illegal) instead of price fixing
Post reply on HN