Earlier quoted context omitted.
This must be on API costs, not counting the $100/200 tiers, right?
yes; fyi usage limits on the $200 claude sub correspond to at least $1.2k/week in api tokens
More than that using Fable!
Opus 5 is currently #1 on Artificial Analysis Intelligence Leaderboard
241–250 of 251 posts
Re: Opus 5 is currently #1 on Artificial Analysis Intelligence Leaderboard
#242The funny thing is that these leaderboards have become completely meaningless for end-users to make decisions on when to use what model. A single metric ranking is useless because each model has strengths and weaknesses for specific domains and tasks. There is no "one best model" anymore, and you might not even need the best model for the level of complexity for your task. For e.g. you might use Fable for UI design,…
A year ago a new Best Model came out, and I was very excited. I tested it on a simple programming task (which required making 3 trivial changes in 3 files). The model did fine. Then I tested its little brother, the older, smaller variant of the same model. It also did fine, except it did it 3x faster and cost 9x less. In this moment, andai was enlightened.
I agree and this is why I think open models will win in the end. There is just so much to gain on being 10% behind the curve. Especially when the curve is far beyond your needs.
Re: Opus 5 is currently #1 on Artificial Analysis Intelligence Leaderboard
#243Earlier quoted context omitted.
Just go to /config. The very second configuration item is “Switch models when a message is flagged” and presumably you want to turn this off. Oh but then you said you never pay Anthropic so you haven’t actually used Claude Code yet. Why would anyone listen to the opinion of a non-user?
Not a great line of thought in general, sometimes the people who aren't doing the thing are the only ones worth listening to. "You aren't repeatedly slamming your head against the wall. Why would anyone listen to the opinion of a non-wall-head-slammer about the merits of wall-head-slamming?"
I don’t really fully trust people’s opinions unless they have first-hand experience. So I would trust someone who has tried the thing and found it to be bad idea, much more than someone who hasn’t tried it and is merely repeating common tropes or talking points.
Re: Opus 5 is currently #1 on Artificial Analysis Intelligence Leaderboard
#244Earlier quoted context omitted.
"completely meaningless" "the only purpose" There's some kernel of truth to what you are saying, but hyperboles like this just aren't accurate. All statistics lie but its better than being blind... What your post really says is that benchmarks only show an average over multiple tasks. Yes, obviously, the point of a statistic is to summarize.
>All statistics lie but its better than being blind disagree
Well, that statement wasn't meant to be taken literally. Maybe not the best wording, given the context...
Re: Opus 5 is currently #1 on Artificial Analysis Intelligence Leaderboard
#245The funny thing is that these leaderboards have become completely meaningless for end-users to make decisions on when to use what model. A single metric ranking is useless because each model has strengths and weaknesses for specific domains and tasks. There is no "one best model" anymore, and you might not even need the best model for the level of complexity for your task. For e.g. you might use Fable for UI design,…
A year ago a new Best Model came out, and I was very excited. I tested it on a simple programming task (which required making 3 trivial changes in 3 files). The model did fine. Then I tested its little brother, the older, smaller variant of the same model. It also did fine, except it did it 3x faster and cost 9x less. In this moment, andai was enlightened.
Claude Code lets you spawn sub-agents mid-session, and I've found framing it adversarially works much better than consensus, and I don't think any of these leaderboards capture it.
Re: Opus 5 is currently #1 on Artificial Analysis Intelligence Leaderboard
#246Earlier quoted context omitted.
I think on swebench verified luna was only like 3% points lower for 1/5 the cost Like 96% vs 93% or something
There is a blog post waiting to be written (that I won't write) about the size/effort tradeoffs, and particularly how small models get some surprisingly good results with lots of turns and reasoning. DeepSWE will let you chart turns taken or tokens used, and FrontierCode will chart tokens. If you use that, you can see Sol high and Terra max get about the same DeepSWE number, but Terra max takes twice the turns. Luna…
Forgot about the ol "but how many tokens did you spend to get _there_"--wish benchmarks would include the number of input/output tokens to achieve the score. I think the closest is Android Bench https://developer.android.com/bench although best you can do is extrapolate off time/cost (iirc they claim to prefer using provider's native API)
In general, smart models work fine with any tools, dumber models need better tools to achieve same results but better tools can eat more context/take more turns
I've gotten decent results with Llama 3.1 8b on Hugging Face tester with Exa MCP since it seems to dump sufficient context into WebSearch/WebFetch type calls even a crappy model almost always gets back what it needs as long as it calls the tool at least once. I had Claude Code look at previous sessions with SearXNG vibe MCP compared to Exa MCP and results got better when it modified SearXNG to work very similarly to Exa. Ended up with this https://gist.github.com/nijave/604c43e3e0fdcd60f5280d3a6b109... although it's really only optimized for "search" not "fetch" at this point. Fetch is basic niquests without Javascript or anything clever
Re: Opus 5 is currently #1 on Artificial Analysis Intelligence Leaderboard
#247Earlier quoted context omitted.
A year ago a new Best Model came out, and I was very excited. I tested it on a simple programming task (which required making 3 trivial changes in 3 files). The model did fine. Then I tested its little brother, the older, smaller variant of the same model. It also did fine, except it did it 3x faster and cost 9x less. In this moment, andai was enlightened.
Claude Code lets you spawn sub-agents mid-session, and I've found framing it adversarially works much better than consensus, and I don't think any of these leaderboards capture it.
What do you mean by adversarially here?
Re: Opus 5 is currently #1 on Artificial Analysis Intelligence Leaderboard
#248Earlier quoted context omitted.
A year ago a new Best Model came out, and I was very excited. I tested it on a simple programming task (which required making 3 trivial changes in 3 files). The model did fine. Then I tested its little brother, the older, smaller variant of the same model. It also did fine, except it did it 3x faster and cost 9x less. In this moment, andai was enlightened.
I agree and this is why I think open models will win in the end. There is just so much to gain on being 10% behind the curve. Especially when the curve is far beyond your needs.
Yeah many tasks are already saturated. For small interactive code changes (I call this power coding like power armor), we got there about a year, maybe year and a half ago — with the small models of the time.
And small models have the advantage that they tend to be much faster. This is an interesting factor because, well speed is always a nice-to-have, but if you go from 10 seconds per turn to one second per turn, the activity actually becomes interactive.
It becomes a fundamentally different way of working. You stay active and engaged the whole time. And also because you are "driving", your mental model does not be synchronized from the code base. So you don't need to spend extra time later catching up.
Re: Opus 5 is currently #1 on Artificial Analysis Intelligence Leaderboard
#249Earlier quoted context omitted.
Yes. I do wish there were benchmarks for specific tech stacks. I.e, if I have an Elixir/Phoenix project, which model performs best (idiomatic, etc.) in 2026? Of course it will be somewhat subjective. And I can hang around those communities for opinions. But it might be useful in a world where it's impractical to constantly compare them all, and it varies pretty widely.
This absolutely should be a thing, but it'll have to be a per-community thing, them building their own dataset and creating their own evals (similar to how people do for production workloads). Although: "(idiomatic, etc.)" I don't think that should be part of the aim (or it should be under-weighted), because... you can just provide guidance on how to do things more idiomatically, rather than depend on that knowledge…
> you can just provide guidance on how to do things more idiomatically, rather than depend on that knowledge already being encoded in the model
I spent some time a while ago comparing various Claude models to various Chinese alternatives. The main difference I noticed was that Claude's code tended to be much more readable and much more elegant.
I found myself asking Sonnet to clean up code written by other models on a very regular basis. (Maybe that's subjective, but I have a feeling it would win on some objective measures of code complexity as well.)
Eventually I realized I could just prompt the Chinese models to give me elegant code. If you don't tell them that they will give you pretty mediocre code by default. (This has improved a little with recent releases.)
I thought that was pretty funny, but then I remembered, it used to be that way with image generation models too!
Back in the day, you had to add "+good -bad" to the prompt, otherwise they wouldn't know you wanted the art to look good, and they would give you all sorts of crap! (Because they've been trained on all sorts of crap. Well, most of the art online is pretty crap, and that goes for most code too.)
Re: Opus 5 is currently #1 on Artificial Analysis Intelligence Leaderboard
#250Earlier quoted context omitted.
It is breaking competition. Would you like all products everywhere be priced like their producers want?
You seem terribly confused. Manufacturers are almost always able to set prices. That is not anti-competitive because it does not imply they are colluding with their competition...
No manufacturers might suggest prices, they can't set them, not in any sensible country.
If they do, they do it secretly and when governments find about that they are going to receive a big fine, together with shops that colluded with them.