This one is different. IYKYK...
Claude Opus 4.5
331–340 of 525 posts
Re: Claude Opus 4.5
#332The burying of the lede here is insane. $5/$25 per MTok is a 3x price drop from Opus 4. At that price point, Opus stops being "the model you use for important things" and becomes actually viable for production workloads. Also notable: they're claiming SOTA prompt injection resistance. The industry has largely given up on solving this problem through training alone, so if the numbers in the system card hold up under a…
3x price drop almost certainly means Opus 4.5 is a different and smaller base model than Opus 4.1, with more fine tuning to target the benchmarks. I'll be curious to see how performance compares to Opus 4.1 on the kind of tasks and metrics they're not explicitly targeting, e.g. eqbench.com
We already know distillation works pretty well. So definitely would make sense Opus 4.5 is effectively smaller (like someone else said, could be via MoE or some other technique too).
We know the big labs are chasing efficiency cans where they can.
Re: Claude Opus 4.5
#333Earlier quoted context omitted.
I like that for this brief moment we actually have a competitive market working in favor of consumers. I ditched my Claude subscription in favor of Gemini just last week. It won't be great when we enter the cartel equilibrium.
Literally "cancelled" my Anthropic subscription this morning (meaning disabled renewal), annoyed hitting Opus limits again. Going to enable billing again. The neat thing is that Anthropic might be able to do this as they massively moving their models to Google TPUs (Google just opened up third party usage of v7 Ironwood, and Anthropic planned on using a million TPUs), dramatically reducing their nvidia-tax spend. Whi…
The behavioral modeling is the product
Re: Claude Opus 4.5
#334Earlier quoted context omitted.
I really don’t understand the hype around Gemini. Opus/Sonnet/GPT are much better for agentic workflows. Seems people get hyped for the first few days. It also has a lot to do with Claude code and Codex.
with gemini you have to spend 30 minutes deleting hundreds of useless comments littered in the code that just describe what the code itself does
The comments would improve code quality because it's a way for the LLM to use a scratchpad to perform locally specific reasoning before writing the proceeding code block, which would be more difficult for the LLM to just one shot.
You could write a postprocessing script to strip the comments so you don't have to do it manually.
Re: Claude Opus 4.5
#335I've played around with Gemini 3 Pro in Cursor, and honestly: I find it to be significantly worse than Sonnet 4.5. I've also had some problems that only Claude Code has been able to really solve; Sonnet 4.5 in there consistently performs better than Sonnet 4.5 anywhere else. I think Anthropic is making the right decisions with their models. Given that software engineering is probably one of the very few domains of AI…
My workflow was usually to use Gemini 2.5 Pro (now 3.0) for high-level architecture and design. Then I would take the finished "spec" and have Sonnet 4.5 perform the actual implementation.
What specific output would you ask Gemini to create for Sonnet? Thanks in advance!
Re: Claude Opus 4.5
#336Earlier quoted context omitted.
3x price drop almost certainly means Opus 4.5 is a different and smaller base model than Opus 4.1, with more fine tuning to target the benchmarks. I'll be curious to see how performance compares to Opus 4.1 on the kind of tasks and metrics they're not explicitly targeting, e.g. eqbench.com
Why? They just closed a $13B funding round. Entirely possible that they're selling below-cost to gain marketshare; on their current usage the cloud computing costs shouldn't be too bad, while the benefits of showing continued growth on their frontier models is great. Hell, for all we know they may have priced Opus 4.1 above cost to show positive unit economics to investors, and then drop the price of Opus 4.5 to spur…
Nobody subsidizes LLM APIs. There is a reason to subsidize free consumer offerings: those users are very sticky, and won't switch unless the alternative is much better.
There might be a reason to subsidize subscriptions, but only if your value is in the app rather than the model.
But for API use, the models are easily substituted, so market share is fleeting. The LLM interface being unstructured plain text makes it simpler to upgrade to a smarter model than than it used to be to swap a library or upgrade to a new version of the JVM.
And there is no customer loyalty. Both the users and the middlemen will chase after the best price and performance. The only choice is at the Pareto frontier.
Likewise there is no other long-term gain from getting a short-term API user. You can't train out tune on their inputs, so there is no classic Search network effect either.
And it's not even just about the cost. Any compute they allocate to inference is compute they aren't allocating to training. There is a real opportunity cost there.
I guess your theory of Opus 4.1 having massive margins while Opus 4.5 has slim ones could work. But given how horrible Anthropic's capacity issues have been for much of the year, that seems unlikely as well. Unless the new Opus is actually cheaper to run, where are they getting the compute from for the massive usage spike that seems inevitable.
Re: Claude Opus 4.5
#337Earlier quoted context omitted.
Last year’s model were at 50-60% on SWE bench-verified actually
I see 25-29% here https://www.swebench.com/viewer.html for models released in Nov 2024 albeit not verified. gpt4o (Aug 2024) was 33% for swe bench verified. Important point because people have a bias to underestimate the speed of ai progress.
Do you people think nobody calls your bluff?
Here’s the launch card of the sonnet 3.5 from a year and a month ago. Guess the number. Ok, Ill tell you: 49.0%. So yeah, the comment you replied to was not really off.
Re: Claude Opus 4.5
#338The LLMs rate of improvement has really slowed down. This looks like a minor improvement in terms of accuracy and big gains from efficiency.
14 months ago we had GPT-4 and now we have models that can get a gold medal at the IMO.
But sure, if you curve fit to the last 3 months you could say things are slowing down, but that's hyper fixating on a very small amount of information.
Re: Claude Opus 4.5
#339Earlier quoted context omitted.
There are two possible explanations for this behavior: the model nerf is real, or there's a perceptual/psychological shift. However, benchmarks exist. And I haven't seen any empirical evidence that the performance of a given model version grows worse over time on benchmarks (in general.) Therefore, some combination of two things are true: 1. The nerf is psychologial, not actual. 2. The nerf is real but in a way that…
> The nerf is psychologial, not actual Once I tested this, I gave the same task for a model after the release and a couple weeks later. In the first attempt it produced a well-written code that worked beautifully, I started to worry about the jobs of the software engineers. Second attempt was a nightmare, like a butcher acting as a junior developer performing a surgery on a horse. Is this empirical evidence? And this…
https://en.wikipedia.org/wiki/Regression_toward_the_mean
The way this works is:
1) x% of users have an exceptional first experience by chance. Nobody who has a meh first experience bothers to try a second time. 2) x²% of users also have an exceptional second experience by chance 3) So a lot of people with a great first experience think the model started off great and got suddenly worse
Suppose it's 25% that have a really great first experience. 25% of them have a great second experience too, but 75% of them see a sudden decline in quality and decide that it must be intentional. After the third experience this population gets bigger again.
So by pure chance and sampling biases you end up convincing a bunch of people that the model used to be great but has gotten worse, but a much smaller population of people who thought it was terrible but got better because most of them gave up early.
This is not in their heads- they really did see declining success. But they experienced it without any changes to the model at all.
Re: Claude Opus 4.5
#340this is very impressive! as much as I love Claude though, is it just me or their limit is much lower compared to others (Gemini and GPT)? At the moment I'm subscribed to Google One AI ($20) which gives me the most value with the 2tb google drive and Cursor ($20). I've subscribed to GPT and Claude as well in the past, I find that I was hitting the limit much faster in Claude compared to all the others, it made me reluctant to subscribe again. from the blog post it seems like they've been prioritising the Max users most of the time?