Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

$500 GPU outperforms Claude Sonnet on coding benchmarks

github.com

Post reply on HN
Print view View original

241–250 of 311 posts

Re: $500 GPU outperforms Claude Sonnet on coding benchmarks

#241

Post by dahart »

Epholys wrote: post #117
weird-eye-issue wrote: post #70

Earlier quoted context omitted.

Why is that? The $200 per month subscription comes with a ton of usage. Opus 4.6 is available on the $20 plan too

> The $200 per month subscription comes with a ton of usage. $200 dollars + VAT is half of my rent. I know HN is not a good place to rant on this subject, but I'm often flabbergasted about the number of people here that lives in a bubble with regard to the price of tech. Or just prices in general. I remember someone who said a few years ago (I'm paraphrasing): "You could just use one of the empty room in your house!"…

$200/mo is a lot, sure, but the shocking part of that comparison is your rent. I didn’t know $400/mo apartments still existed. For most people in the US and EU, $200 would be closer to 15%-20% of rent I think? My cell phone bill for my family is almost $200/mo.

Last year, at first, $200 seemed crazy. Now that I’m getting addicted to coding agents, not so much. Some companies are paying API rates for AI for employees, and it’s a lot more than $200/mo. It seems like funny money, and I’m not sure it’ll last.

Re: $500 GPU outperforms Claude Sonnet on coding benchmarks

#242

Post by Filligree »

throwaway173738 wrote: post #201
mememememememo wrote: post #183

Earlier quoted context omitted.

Thanks for the alternative perspective. I think I am in the middle. I can afford $200/m but it'd be a brainer. And I don't pay that as I barely use home AI enough to warrant it. I am also amazed at the richer end of HN but now I realize I am priviledged. Earned it? Like fuck I did. Lucky to be born a geek in late 20c. I'd be useless as a middle ages guy.

If I found myself in the middle ages I’d just become a blacksmith or a miller.

Do you have the genetics for that? It takes a lot of raw strength, and not that much intelligence.

Re: $500 GPU outperforms Claude Sonnet on coding benchmarks

#243

Post by andoando »

weird-eye-issue wrote: post #70
rf15 wrote: post #67

Earlier quoted context omitted.

You cannot afford the SOTA.

Why is that? The $200 per month subscription comes with a ton of usage. Opus 4.6 is available on the $20 plan too

I dunno how you guys even go throuh the $200 subscription. I use it every day for work and side projects doing tasks in parallel and Im no where newr the limit on $100.

Re: $500 GPU outperforms Claude Sonnet on coding benchmarks

#244

Post by dakolli »

bartread wrote: post #163
bloppe wrote: post #76

Generating big chunks of code is rarely what I want from an agent. They really shine for stuff like combing through logs or scanning dozens of source files to explain a test failure. Which benchmark covers that? I want the debugging benchmark that tests mastery of build systems, CLIs, etc.

I agree. Also good for small changes that need to be applied consistently across an entire codebase. I recently refactored our whole app from hard deletes to soft deletes. There are obviously various ways to skin this particular cat, but the way I chose needed all our deletions updated and also needed queries updating to exclude soft deleted rows, except in specific circumstances (e.g., admins restoring accidentally…

must be something incredibly simple you're making out more complicated than it actually is, I've never seen an LLM do these things well.

Re: $500 GPU outperforms Claude Sonnet on coding benchmarks

#245

Post by Aurornis »

This AI-written project is running its own LiveCodeBench on a completely different methodology. The AI-written notes even admit it:

> ATLAS scores are from 599 LCB tasks using the full V3 pipeline (best-of-3 + Lens selection + iterative repair) on a frozen 14B quantized model or "pass@k-v(k=3)". Competitor scores are single-shot pass@1 (zero-shot, temperature 0) from Artificial Analysis on 315 LCB problems -- not the same task set, so this is not a controlled head-to-head.

Instead of following the LiveCodeBench methodology, it's a harness that spins up a sandbox and spends a long time testing and refining the solution. If you did the same for Sonnet, GPT5.4, or other models they would also get significantly higher scores and they'd do it faster.

The AI-coded README is also full of signs of vibecoded slop like the discoveries that some of the complex structures implemented were not actually being used or contributing anything to the output.

Re: $500 GPU outperforms Claude Sonnet on coding benchmarks

#246

Post by bartread »

CraigJPerry wrote: post #170
bartread wrote: post #163

Earlier quoted context omitted.

I agree. Also good for small changes that need to be applied consistently across an entire codebase. I recently refactored our whole app from hard deletes to soft deletes. There are obviously various ways to skin this particular cat, but the way I chose needed all our deletions updated and also needed queries updating to exclude soft deleted rows, except in specific circumstances (e.g., admins restoring accidentally…

Do you not end up breaking half the value of referential integrity doing it that way (e.g. you had to update all the queries but now you have a sharp edge in that all future queries need to remember to be soft delete aware. Not a blocker for sure, just a sharp edge). You know your system better than me for sure, a random commenter on a website :-D your comment just shocked me out of my daze enough for my brain to say…

Yeah, I did consider moving records to shadow tables, but - because of the nature of our data - it requires moving a lot of child records as well, so it's quite a lot of additional churn in WAL, and the same for restore. And this approach has its own challenges with referential integrity.

More than that, though: lots of queries for reporting, and the like, suddenly need to use JOINs. Same for admin use cases where we want them to be able to see archived and live data in a unified view. The conclusion I came to is it doesn't really eliminate complexity for us: just moves it elsehwere.

Totally valid approach though. I'd also considered different views for live versus archived (or live+archived) data. Again, it solves some issues, but moves complexity elsewhere.

The other key point: it's a Ruby on Rails system so the moment you start doing funky stuff with separate tables or views, whilst it is doable, you lose a lot of the benefits of Active Record and end up having to do a lot more manual lifting. So, again, this sort of played against the alternatives.

As I say, not to diss other approaches: in a different situation I might have chosen one of them.

My conclusion - not for the first time - is that soft delete obviously adds some level of irreducible complexity to an application or system versus hard delete no matter how you do it. Whether or not that extra complexity is worth it very much depends on the application and your user/customer base.

For some people, just the ability to restore deleted rows from backup would be enough - and in other cases it's been enough for me - but that is always a bit of a faff so not a great fit if you're optimising for minimal support overhead and rapid turnaround of any issues that do arise.

Re: $500 GPU outperforms Claude Sonnet on coding benchmarks

#247

Post by bartread »

dakolli wrote: post #244
bartread wrote: post #163

Earlier quoted context omitted.

I agree. Also good for small changes that need to be applied consistently across an entire codebase. I recently refactored our whole app from hard deletes to soft deletes. There are obviously various ways to skin this particular cat, but the way I chose needed all our deletions updated and also needed queries updating to exclude soft deleted rows, except in specific circumstances (e.g., admins restoring accidentally…

must be something incredibly simple you're making out more complicated than it actually is, I've never seen an LLM do these things well.

This is what gives me the warm fuzzies about the HN community: people jumping to wild conclusions about your domain and systems based on a 4 sentence comment. /s

Re: $500 GPU outperforms Claude Sonnet on coding benchmarks

#248

Post by alifeinbinary »

selcuka wrote: post #10

It's a race to the bottom. DeepSeek beats all others (single-shot), and it is ~50% cheaper than the cost of local electricity only. > DeepSeek V3.2 Reasoning 86.2% ~$0.002 API, single-shot > ATLAS V3 (pass@1-v(k=3)) 74.6% ~$0.004 Local electricity only, best-of-3 + repair pipeline

All those parameters and it still won't answer questions about Tianenman Square in 1989... :(

Re: $500 GPU outperforms Claude Sonnet on coding benchmarks

#249

Post by 1123581321 »

Epholys wrote: post #117
weird-eye-issue wrote: post #70

Earlier quoted context omitted.

Why is that? The $200 per month subscription comes with a ton of usage. Opus 4.6 is available on the $20 plan too

> The $200 per month subscription comes with a ton of usage. $200 dollars + VAT is half of my rent. I know HN is not a good place to rant on this subject, but I'm often flabbergasted about the number of people here that lives in a bubble with regard to the price of tech. Or just prices in general. I remember someone who said a few years ago (I'm paraphrasing): "You could just use one of the empty room in your house!"…

It’s a good reminder. Claude Max costs about as much as the global poverty line ($3/day.) I think it’s okay to invest in it, but we should try to make sure it’s worthwhile, and also invest in charity.

Re: $500 GPU outperforms Claude Sonnet on coding benchmarks

#250

Post by sroussey »

TeMPOraL wrote: post #162
Bombthecat wrote: post #124

Earlier quoted context omitted.

That's why ai is for the "rich". Poor people or later on middle class will be left behind....

Nah, that's why you cannot not afford the subscriptions these days. Whatever your needs, ever since Claude Code became a thing, subscription costs come out massively cheaper than pay-as-you-go per-token API pricing. Also SOTA models are so much better than anything else, that using older or open models will just cost you more in tokens/electricity than going for SOTA subscription. Subscriptions are definitely middle-…

The subscriptions are purposely sold for less than cost. The subsidy will end some day.
Post reply on HN