Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Opus 5 is currently #1 on Artificial Analysis Intelligence Leaderboard

artificialanalysis.ai

Post reply on HN
Print view View original

41–50 of 251 posts

Re: Opus 5 is currently #1 on Artificial Analysis Intelligence Leaderboard

#41

Post by gck1 »

andy99 wrote: post #19

#1 in a very close race is way less useful when you have to walk on eggshells to avoid triggering censorship (“safeguards”) that either refuse or knock it down to another model. I’ve almost completely stopped using Claude (except some legacy workflows) for this reason, reliability matters more than scoring 61 instead of 57. To me Claude is the most compromised and unreliable model (between the censorship and the id c…

Every time an online chatter (e.g. "limits are better", "model is better") makes me to reevaluate my principle of never paying Anthropic, I go to the model card, which strengthens my belief in the principle.

Why is Anthropic is so hell-bent on this auto/silent downgrade? Do they have a single user who prefers an auto-lobotomization instead of a refusal? Have they learned nothing from the backlash the first time?

Re: Opus 5 is currently #1 on Artificial Analysis Intelligence Leaderboard

#42

Post by mchusma »

firasd wrote: post #14

Very interesting that one of the components is "AA-Omniscience Index" AA-Omniscience Index (higher is better) measures knowledge reliability and hallucination. It rewards correct answers, penalizes hallucinations, and has no penalty for refusing to answer. This seems to be a good proxy for param size/density and the ranking breaks down as such: Claude Fable 5 (with fallback), Gemini 3.1 Pro Preview, Claude Opus 5 (Ma…

Gemini 3.1 pro is really good for knowledge tasks. Google has done well there. And image analysis with Gemini flash 3.6 is solid. It’s just anything coding or agentic they fall short.

Re: Opus 5 is currently #1 on Artificial Analysis Intelligence Leaderboard

#43

Post by chronogram »

firasd wrote: post #14

Very interesting that one of the components is "AA-Omniscience Index" AA-Omniscience Index (higher is better) measures knowledge reliability and hallucination. It rewards correct answers, penalizes hallucinations, and has no penalty for refusing to answer. This seems to be a good proxy for param size/density and the ranking breaks down as such: Claude Fable 5 (with fallback), Gemini 3.1 Pro Preview, Claude Opus 5 (Ma…

I'm not so sure. Especially with 3.6 Flash being 24 to Sol's 22. The 3.x Flash models were thought to fit on a single TPU 8i and it's the odd one out in that list at a whopping 234.7t/s compared to Sol's 64.4t/s and Opus's 56.3t/s. Even 3.1 Pro is a much higher 113.9t/s than the others.

I think AA-Omniscience Accuracy follows your expectations better. An ultra size Fable at 61%, followed by large frontier models like Sol, 5.5 and Opus. With Flash being up there. I assume because Gemini is more focused on general knowledge to operational cost in particular, rather than getting the highest scores in coding benchmarks. If you go to Domain Score (Normalized) you'll see that the Gemini models are only less competitive in Software. And that's where Sol goes from 6 in Health to 71 in Software.

Re: Opus 5 is currently #1 on Artificial Analysis Intelligence Leaderboard

#44

Post by wmf »

charcircuit wrote: post #18
impulser_ wrote: post #11

Earlier quoted context omitted.

It shouldn't be surprising OpenAI does have the most compute out of all the major labs. The only reason why Anthropic models are expensive is they are the most in demand models in the world and Anthropic is fighting for compute. The only way to you limit demand for your model is increasing API pricing this is also why Anthropic probably has great margin and probably is profitable compared to OpenAI.

I also suspect there is a price fixing agreement between all of the inference providers for Claude (such as Amazon, Anthropic, Microsoft, etc).

"Price fixing" isn't the correct term here but yes, it's very common to have the same price across different retailers/resellers.

Re: Opus 5 is currently #1 on Artificial Analysis Intelligence Leaderboard

#45

Post by weird-eye-issue »

afavour wrote: post #22
andy99 wrote: post #19

#1 in a very close race is way less useful when you have to walk on eggshells to avoid triggering censorship (“safeguards”) that either refuse or knock it down to another model. I’ve almost completely stopped using Claude (except some legacy workflows) for this reason, reliability matters more than scoring 61 instead of 57. To me Claude is the most compromised and unreliable model (between the censorship and the id c…

What are you asking that you’re so regularly running into censorship?

Literally anything related to nutrition, athletic performance, etc especially if you ask it for research or sources

Re: Opus 5 is currently #1 on Artificial Analysis Intelligence Leaderboard

#47

Post by gck1 »

afavour wrote: post #22
andy99 wrote: post #19

#1 in a very close race is way less useful when you have to walk on eggshells to avoid triggering censorship (“safeguards”) that either refuse or knock it down to another model. I’ve almost completely stopped using Claude (except some legacy workflows) for this reason, reliability matters more than scoring 61 instead of 57. To me Claude is the most compromised and unreliable model (between the censorship and the id c…

What are you asking that you’re so regularly running into censorship?

Reverse engineering. Codex sometimes displays an advisory prompt when classifier trips - "Wait longer while we evaluate this request further or use a dumber model". If you do nothing, it'll just take some time and almost always succeed.

It does require some brainwashing of the model to get it to the state where model itself agrees to do RE work though. But at least it's all predictable.

Re: Opus 5 is currently #1 on Artificial Analysis Intelligence Leaderboard

#48

Post by markasoftware »

gck1 wrote: post #41
andy99 wrote: post #19

#1 in a very close race is way less useful when you have to walk on eggshells to avoid triggering censorship (“safeguards”) that either refuse or knock it down to another model. I’ve almost completely stopped using Claude (except some legacy workflows) for this reason, reliability matters more than scoring 61 instead of 57. To me Claude is the most compromised and unreliable model (between the censorship and the id c…

Every time an online chatter (e.g. "limits are better", "model is better") makes me to reevaluate my principle of never paying Anthropic, I go to the model card, which strengthens my belief in the principle. Why is Anthropic is so hell-bent on this auto/silent downgrade? Do they have a single user who prefers an auto-lobotomization instead of a refusal? Have they learned nothing from the backlash the first time?

I don't believe they do silent downgrades right now, they're loud about it.

Re: Opus 5 is currently #1 on Artificial Analysis Intelligence Leaderboard

#49

Post by nu11ptr »

I don't have a horse in this race, but to me this makes GPT-5.6 Sol Max look better. It is about half the cost for nearly the exact same performance. It just goes to show how expensive Fable really is when Opus 5 is still this expensive relative to GPT 5.6.

Re: Opus 5 is currently #1 on Artificial Analysis Intelligence Leaderboard

#50

Post by kccqzy »

gck1 wrote: post #41
andy99 wrote: post #19

#1 in a very close race is way less useful when you have to walk on eggshells to avoid triggering censorship (“safeguards”) that either refuse or knock it down to another model. I’ve almost completely stopped using Claude (except some legacy workflows) for this reason, reliability matters more than scoring 61 instead of 57. To me Claude is the most compromised and unreliable model (between the censorship and the id c…

Every time an online chatter (e.g. "limits are better", "model is better") makes me to reevaluate my principle of never paying Anthropic, I go to the model card, which strengthens my belief in the principle. Why is Anthropic is so hell-bent on this auto/silent downgrade? Do they have a single user who prefers an auto-lobotomization instead of a refusal? Have they learned nothing from the backlash the first time?

Just go to /config. The very second configuration item is “Switch models when a message is flagged” and presumably you want to turn this off.

Oh but then you said you never pay Anthropic so you haven’t actually used Claude Code yet. Why would anyone listen to the opinion of a non-user?

Post reply on HN