Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Claude Opus 4.8

anthropic.com

Post reply on HN
Print view View original

261–270 of 1001 posts

Re: Claude Opus 4.8

#261

Post by Bnjoroge »

aronowb14 wrote: post #85
onlyrealcuzzo wrote: post #13

Does anyone troll these releases and cherry pick random metrics other companies would cherry pick to show how amazing their models are? There's like 8 million benchmarks. Every release, every model randomly picks 5-10 where they win in everything except 1, to make it look like they aren't randomly cherry picking benchmarks they probably benchmaxxed for.

https://arena.ai/leaderboard - I’ve found this company is a pretty good ranker - not sure their exact methodology but during day to day programming with Claude / gpt models I’ve felt qualitatively what they report

Have you seen https://deepswe.datacurve.ai/blog? This is the closest to a vibe check i’ve felt even with the open models.

Re: Claude Opus 4.8

#262

Post by XCSme »

On my tests[0] it does a bit worse, and it's almost 2x expensive than Opus 4.7...

I was surprised to see that it failed a Data extraction test (it gets it right 2/3 times, but one time it randomly returns null for a value instead).

It makes sense a bit that it fails more Trivia/Domain-specific knowledge tasks (I think models are more and more trained towards agentic use-case than general intelligence).

[0]: https://aibenchy.com/compare/anthropic-claude-opus-4-7-mediu...

Re: Claude Opus 4.8

#263

Post by ddosmax556 »

lxxpxlxxxx wrote: post #254

My experience with these new releases is that the gains in performance are negated by the price increases and it seems like: Performance gains: 1.2x Price increases: 1.8x

They're not negated, smarter is smarter, but you have to reach deeper in your pocket. I think this will happen more and more - the smartest models get more expensive. But it won't matter - the current models we have today will get cheaper and can still be used for what they're used today.

Re: Claude Opus 4.8

#264

Post by firebirdn99 »

onlyrealcuzzo wrote: post #136
NiloCK wrote: post #77

A rambling comment: I think this is the first time we've had a third minor version bump on a frontier Anthropic model. (I count the 0.5s as major here, because they've been issued non-sequentially and also corresponded to massive capability leaps, eg, Sonnet 3.5, Opus 4.5). So now the Opus 4.5 family has successors 4.6, 4.7, and 4.8, each posting fairly modest claimed gains. My own experience w/ 4.6 and 4.7 are that…

I won't be surprised if the next gen frontier models are the last. There's orders of magnitude of low hanging juice to squeeze out of smaller models. It is almost guaranteed that a 60-90B model can outperform current SOTA in coding tasks within 2-3 years (design not certain, probably unlikely). It is far less clear that a 1.2T model will be meaningfully better enough to justify training it. As far as reasoning is con…

you just need to look at Mythos to see the jump in performance from a 10T(?) model. As they scale, they get more capable. We might have an yearly release, but I believe the releases will continue, as long as scaling laws are in tact, and there's huge problems still need solving. (think cancer)

Re: Claude Opus 4.8

#266

Post by simplyluke »

raincole wrote: post #154
wg0 wrote: post #104

There is a hole in the boat's bottom due to Chinese models. They might not be as good but they are not bad either or at least I had hard time finding any issues with Deepseekv4 Flash and Pro variants. They get their job done sometimes rarely giving up till they are done what they are after. So even for enterprise deployments, as the dust settles down, CFO/CTOs might find out that deploying on an internal cluster of G…

I had been saying this on HN repeatedly: people are going to use the smartest models for coding. They don't care how cheap your tokens are if they don't have the highest probability of solving your programming tasks. And I was dead wrong. Now I mostly use DeepSeek Pro myself.

The other thing that's changing is more and more CFOs are looking at the AI spend in engineering departments and hitting the brakes. Token leaderboards were cool when the spend wasn't a double-digit-percent of the entire department's budget including salaries.

Re: Claude Opus 4.8

#267

Post by MattRogish »

gslepak wrote: post #58

On page 102 of the system card [1] I'm pleased to see evaluation against "creative mastery". In our work we asked several frontier AIs to come up with an API we needed. We compared Opus 4.7 and GPT-5.5 (among others). Opus 4.7 came up with the most creative and intelligent API design that pleasantly surprised us, especially given that GPT-5.5 was passing it on various coding benchmarks. What I noticed is that we don'…

Agreed, my vibes tell me 4.6 is a better coder than 4.7. 4.7 is a much better strategic thinker and maintains overall "better architecture" than 5.5. 5.5 is way better than either at coding, but more expensive. So I have 4.7 do the planning/architecture, 4.6 does the coding, then 5.5 critiques and fixes it.

Re: Claude Opus 4.8

#268

Post by jpadkins »

bel8 wrote: post #123
onlyrealcuzzo wrote: post #13

Does anyone troll these releases and cherry pick random metrics other companies would cherry pick to show how amazing their models are? There's like 8 million benchmarks. Every release, every model randomly picks 5-10 where they win in everything except 1, to make it look like they aren't randomly cherry picking benchmarks they probably benchmaxxed for.

On this note, is there a benchmark aggregator to compile all benchmarks in a single large grid?

I find this site useful https://artificialanalysis.ai/leaderboards/models

Re: Claude Opus 4.8

#269

Post by ninjagoo »

Philpax wrote: post #35
clutch89 wrote: post #6

> One of the most prominent improvements in Opus 4.8 is its honesty Anthropic talks about their own models as if they're discovering new species in the wild...

AI is grown, not built, and like with anything you grow, you'll never be able to predict exactly how it will turn out.

> AI is grown, not built, and like with anything you grow, you'll never be able to predict exactly how it will turn out.

Remember when the frontier labs found out that curated high-quality training was critical to making better models?

Basically, just like high-quality and more education tends to make better humans, on average, I think we can expect quality education to turn out better ai, on average, and with better repeatability than with humans because of better control over the initial conditions and environment.

Post reply on HN