Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Claude Opus 5

anthropic.com

Post reply on HN
Print view View original

421–430 of 1001 posts

Re: Claude Opus 5

#421

Post by manojlds »

alvis wrote: post #155
postalcoder wrote: post #135

I think the most important thing here is not absolute performance. It's that organizations now have access to a Fable-ish model without Fable's 30-day data retention requirement[0]. > "Consistent with prior Opus models, Opus 5 does not have data retention requirements for general access."[1] On the Opus model release page, the reason why Fable doesn't have an ARC-AGI score is because of that retention policy[2]. 0: h…

Also the cost per task. It appears to be significantly cheaper, cheaper than sonnet!

Opus 4.8 was already shown to be cheaper than Sonnet 5 when Sonnet 5 was released (by Anthropic)

Re: Claude Opus 5

#422

Post by roboyoshi »

not_a9 wrote: post #182
yukIttEft wrote: post #165

Earlier quoted context omitted.

What are your purposes?

Reversing for the most part, though lately I’ve been doing some code obfuscation/binary rewriting stuff. Fable will switch to Opus instantly on these and I’m unsure how this will perform. I suppose the only way to find out is to test.

It's rather broad right now. I started reverse engineering a mac app and it started reading some binary data and then quickly told me to switch to Opus 4.8 to continue, because the guardrails kicked in.

Re: Claude Opus 5

#423

Post by vadansky »

mcbuilder wrote: post #231
artninja1988 wrote: post #58

That's a crazy arc 3 score. What do people think of this? Are models actually developing fluid intelligence like what the creators claim to be measuring? Is it jus do to training for it? Is the benchmark flawed?

Have you played Arc 3? It seems like more of a simple optimization problem (think Sokoban) than anything approaching fluid intelligence. Whether a multi hundred billion dollar company would spend time benchmaxxing a highly publicized benchmark that claims to confer AGI is an exercise left to the reader, but I doubt Claude Plays Pokemon is suddenly going to get past Mt. Doom now.

> Claude Plays Pokemon is suddenly going to get past Mt. Doom now.

I miss him... But for reference he did get past Doom and got pretty far in the strength puzzle too before he cut cut off. He was looping and just brute forcing it.

Re: Claude Opus 5

#424

Post by manojlds »

gonzalohm wrote: post #284
postalcoder wrote: post #135

I think the most important thing here is not absolute performance. It's that organizations now have access to a Fable-ish model without Fable's 30-day data retention requirement[0]. > "Consistent with prior Opus models, Opus 5 does not have data retention requirements for general access."[1] On the Opus model release page, the reason why Fable doesn't have an ARC-AGI score is because of that retention policy[2]. 0: h…

I don't understand how the data retention works. My company has an enterprise license with no data retention but if I ask Claude about past conversations, it remembers. So surely the information is being stored somewhere

Claude Code? It stores a memory.md file.

Re: Claude Opus 5

#425

Post by artemisart »

2001zhaozhao wrote: post #304
Dibes wrote: post #104

I'm not sure what to make of this graph[0]. It shows medium as the most effective thinking mode by far for frontier code. It's the only case that I saw going through the system card where more reasoning effort meaningfully negatively impacted the resulting eval. I know sometimes max efforts show a small dip, but this is substantial. I wonder why in the world that is? [0] https://imgur.com/a/Nv8V7Ry

It is probably from randomness. The benchmark tasks nowadays are so long that you can't really afford to run a large number of samples of them per model & effort combination

No it's a mean of 5 runs.

> We report FrontierCode’s overall score, a composite measure that grades each patch on blocking functional criteria (held-out unit tests) together with weighted code-quality rubric criteria, as mean@5.

They don't explain more in the system card, I guess higher effort levels could loose points on the code quality / scope / style / maintainability stuff?

Re: Claude Opus 5

#426

Post by taybin »

hk__2 wrote: post #343
petilon wrote: post #335

The naming system is so confusing. Is Opus better than Sonnet? Where does Haiku fit in? How can you tell from the name? I can't keep track of all these names or make guesses from the names. Suggestion for a better naming system: use the words "Pro", "Plus", etc.: Claude 5 Pro, Claude 5 Standard, Claude 5 Fast, Claude 5 Mini.

An opus is longer than a sonnet, which is longer than a haiku. Hence Opus > Sonnet > Haiku. > Suggestion for a better naming system: use the words "Pro", "Plus", etc.: Claude 5 Pro, Claude 5 Standard, Claude 5 Fast, Claude 5 Mini. This is not possible: Standard (Free) / Pro / Max are plan names. Fast is a mode.

How long is a fable though?

Re: Claude Opus 5

#427

Post by itissid »

I found opus 4.8 too agreeable and too wordy(as opposed to codex) and too agreeable. If you are reading documents generating by it was too much. TBH. Fable did a bit better on this. Anyone seen a marked difference with opus 5 on this?

Re: Claude Opus 5

#428

Post by pixl97 »

onlyrealcuzzo wrote: post #419
alvis wrote: post #155

Earlier quoted context omitted.

Also the cost per task. It appears to be significantly cheaper, cheaper than sonnet!

I can't believe they released the charts they did. It basically shows that Sol absolutely demolishes Fable at every part of the cost curve for coding for the same level of quality. Opus is competitive. It just has a higher level of quality / higher cost to start.

If fable costs more to run than the markup they still come out ahead.

Re: Claude Opus 5

#429

Post by oblio »

gigatexal wrote: post #238
postalcoder wrote: post #135

I think the most important thing here is not absolute performance. It's that organizations now have access to a Fable-ish model without Fable's 30-day data retention requirement[0]. > "Consistent with prior Opus models, Opus 5 does not have data retention requirements for general access."[1] On the Opus model release page, the reason why Fable doesn't have an ARC-AGI score is because of that retention policy[2]. 0: h…

insane pricing: " Claude Opus 5 is available today on all platforms, priced at $5 per million input tokens and $25 per million output tokens (the same as Opus 4.8)"

Why is it insane if it's the same as the previous version?

Re: Claude Opus 5

#430

Post by jll29 »

tadfisher wrote: post #156
nightpool wrote: post #132

Earlier quoted context omitted.

You're comparing the "score percentage" (e.g. out of the total number of partial points available, how many did the agent achieve) to the "completion percentage" (how many tasks does the model score 100% on). The paper says "Claude Opus 4.8 with maximum thinking and batched tool calls scores best but still completes only 20.6% of tasks at a 54.8% partial score", which is ~the same number that Anthropic reports here (…

In what world is 55.7 the same number as 54.8? What variance is acceptable to publish without a retraction?

I don't know who downvoted the parent or why, but it's a fair question IMHO.

The answer is there can be dramatic difference running a benchmark one time, because LLMs are not deterministic. A proper methodology would ask each question 20 times and calculate the mean correctness across experiments.

The reason is that the temperature parameter introduces random behavior.

Post reply on HN