These small models, having been fine-tuned for the test, achieve frighteningly high scores, yet perform abysmally in real-world scenarios.
$500 GPU outperforms Claude Sonnet on coding benchmarks
161–170 of 311 posts
Re: $500 GPU outperforms Claude Sonnet on coding benchmarks
#162Earlier quoted context omitted.
> The $200 per month subscription comes with a ton of usage. $200 dollars + VAT is half of my rent. I know HN is not a good place to rant on this subject, but I'm often flabbergasted about the number of people here that lives in a bubble with regard to the price of tech. Or just prices in general. I remember someone who said a few years ago (I'm paraphrasing): "You could just use one of the empty room in your house!"…
That's why ai is for the "rich". Poor people or later on middle class will be left behind....
Nah, that's why you cannot not afford the subscriptions these days. Whatever your needs, ever since Claude Code became a thing, subscription costs come out massively cheaper than pay-as-you-go per-token API pricing. Also SOTA models are so much better than anything else, that using older or open models will just cost you more in tokens/electricity than going for SOTA subscription.
Subscriptions are definitely middle-class targeted. $20/month is not much for the value provided, at least not in the western world.
But if by "rich" you just mean "westerners", then in this sense, the same is and has always been true for computing in general.
Re: $500 GPU outperforms Claude Sonnet on coding benchmarks
#163Generating big chunks of code is rarely what I want from an agent. They really shine for stuff like combing through logs or scanning dozens of source files to explain a test failure. Which benchmark covers that? I want the debugging benchmark that tests mastery of build systems, CLIs, etc.
I agree. Also good for small changes that need to be applied consistently across an entire codebase.
I recently refactored our whole app from hard deletes to soft deletes. There are obviously various ways to skin this particular cat, but the way I chose needed all our deletions updated and also needed queries updating to exclude soft deleted rows, except in specific circumstances (e.g., admins restoring accidentally deleted data).
Of course, this is not hard to do manually but is is a bloody chore and tends toward error prone. But the agent made short work of it, for which I was very grateful.
Re: $500 GPU outperforms Claude Sonnet on coding benchmarks
#164Earlier quoted context omitted.
>I'm often flabbergasted about the number of people here that lives in a bubble with regard to the price of tech Sorry, no. You live in the bubble, the people you think are living in a bubble are actually doing the very opposite and taking advantage of the lack of bubbles in our globally connected world. Today, basically anyone can sell any bullshit to billions of people around the world. We’ve never lived in less of…
I guess all those people who live in not-SF just can't be bothered to succeed!
$20/month is not above middle class in most of the world.
$200/month is, but you don't need that for anything except beyond-casual use of coding agents.
Re: $500 GPU outperforms Claude Sonnet on coding benchmarks
#165Earlier quoted context omitted.
The person you were replying to was not talking about SF but you specifically called out SF so you were implying that
The thread started with "$200 is a lot for most of the world", the person I was replying to said "no it's not, now anyone can sell to billions of people", and I said "company success being concentrated in SF shows that that's not true". I didn't say "only SF can afford $200/mo".
"I guess all those people who live in not-SF just can't be bothered to succeed!"
Re: $500 GPU outperforms Claude Sonnet on coding benchmarks
#166Earlier quoted context omitted.
The thread started with "$200 is a lot for most of the world", the person I was replying to said "no it's not, now anyone can sell to billions of people", and I said "company success being concentrated in SF shows that that's not true". I didn't say "only SF can afford $200/mo".
"I guess all those people who live in not-SF just can't be bothered to succeed!"
I explained it in my previous comment, I'm not going to explain it more than that.
Re: $500 GPU outperforms Claude Sonnet on coding benchmarks
#167Earlier quoted context omitted.
Why is that? The $200 per month subscription comes with a ton of usage. Opus 4.6 is available on the $20 plan too
> The $200 per month subscription comes with a ton of usage. 200 USD/month is a number only really affluent programmers (e.g. in the Silicon Valley) can perhaps pay easily.
The $100 already gives plenty of usage and is more than worth it, and I'm definitely not an affluent SV developer. I've only ever hit the 5h limit once in the last month, although I rarely run more than 3 agents at once, and I don't use ridiculously expensive tools like Gas Town.
Re: $500 GPU outperforms Claude Sonnet on coding benchmarks
#168Earlier quoted context omitted.
And there’s an incentive to publish evidence of this to discourage it, do you have any?
Models aren't just big bags of floats you imagine them to be. Those bags are there, but there's a whole layer of runtimes, caches, timers, load balancers, classifiers/sanitizers, etc. around them, all of which have tunable parameters that affect the user-perceptible output.
There really always is a man behind the curtain eh?
Re: $500 GPU outperforms Claude Sonnet on coding benchmarks
#169Earlier quoted context omitted.
"Opus 4.6 is available on the $20 plan too"
Anthropic’s $20 plan gives you such a pittance of tokens that it’s borderline unusable for anything more than a few scripts or a toy app. If $20 is all you have you’d do _much_ better going with chatgpt
The Codex plan for the $20 ChatGPT plan goes much further than Claude's $20 plan, but it's still not enough if you plan to work full-time with it.
Re: $500 GPU outperforms Claude Sonnet on coding benchmarks
#170Generating big chunks of code is rarely what I want from an agent. They really shine for stuff like combing through logs or scanning dozens of source files to explain a test failure. Which benchmark covers that? I want the debugging benchmark that tests mastery of build systems, CLIs, etc.
I agree. Also good for small changes that need to be applied consistently across an entire codebase. I recently refactored our whole app from hard deletes to soft deletes. There are obviously various ways to skin this particular cat, but the way I chose needed all our deletions updated and also needed queries updating to exclude soft deleted rows, except in specific circumstances (e.g., admins restoring accidentally…
Do you not end up breaking half the value of referential integrity doing it that way (e.g. you had to update all the queries but now you have a sharp edge in that all future queries need to remember to be soft delete aware. Not a blocker for sure, just a sharp edge).
You know your system better than me for sure, a random commenter on a website :-D your comment just shocked me out of my daze enough for my brain to say "but I always move the record to another table rather than soft delete" and i felt compelled to give unsolicited and likely wrong opinion.