Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

$500 GPU outperforms Claude Sonnet on coding benchmarks

github.com

Post reply on HN
Print view View original

271–280 of 311 posts

Re: $500 GPU outperforms Claude Sonnet on coding benchmarks

#271

Post by withinboredom »

scrollop wrote: post #264
hhh wrote: post #104

Earlier quoted context omitted.

The models don’t change.

You sure about that? https://marginlab.ai/trackers/claude-code/

Well, I don't see 4.5 on there ... so I'm not sure what you're trying to say.

And today is a 53% pass rate vs. a baseline 56% pass rate. That's a huge difference. If we recall what Anthropic originally promised a "max 5" user https://github.com/anthropics/claude-code/issues/16157#issue... -- which they've since removed from their site...

50-200 prompts. That's an extra 1-6 "wrong solutions" per 5 hours ... and you have to get a lot of wrong answers to arrive at a wrong solution.

Re: $500 GPU outperforms Claude Sonnet on coding benchmarks

#272

Post by NortySpock »

andyferris wrote: post #182
CraigJPerry wrote: post #170

Earlier quoted context omitted.

Do you not end up breaking half the value of referential integrity doing it that way (e.g. you had to update all the queries but now you have a sharp edge in that all future queries need to remember to be soft delete aware. Not a blocker for sure, just a sharp edge). You know your system better than me for sure, a random commenter on a website :-D your comment just shocked me out of my daze enough for my brain to say…

I move the record to another _index_, generally. It depends whether you reliably control all the DB client code, of course.

This, make sure the 'active' flag (or deleted_at timestamp) is part of most indexes and you're probably going to see very small impacts on reads.

It then turns into a slowly-growing problem if you never ever clean up the soft-deleted records, but just being able to gain auditability nearly immediately is usually well worth kicking the can down the road.

Re: $500 GPU outperforms Claude Sonnet on coding benchmarks

#273

Post by TeMPOraL »

sroussey wrote: post #250
TeMPOraL wrote: post #162

Earlier quoted context omitted.

Nah, that's why you cannot not afford the subscriptions these days. Whatever your needs, ever since Claude Code became a thing, subscription costs come out massively cheaper than pay-as-you-go per-token API pricing. Also SOTA models are so much better than anything else, that using older or open models will just cost you more in tokens/electricity than going for SOTA subscription. Subscriptions are definitely middle-…

The subscriptions are purposely sold for less than cost. The subsidy will end some day.

We'll cross that bridge when we come to it. Especially in context of discussing living at different economic strata, customers are neither expected nor supposed to voluntarily overpay out of a belief this will make an industry not try to rugpull everyone at some point.

Re: $500 GPU outperforms Claude Sonnet on coding benchmarks

#274

Post by hyperbovine »

Epholys wrote: post #152
hyperbovine wrote: post #140

Earlier quoted context omitted.

Work pays.

I'm not sure I've correctly understood what you're implying. If it's that I'm not working, well, I'm employed. It it's that I'm not working enough to not have this money... Well, we still go back to the bubble. Not everywhere in the world you can easily find a job that pays you enough, even if you accept to work more. And the employer will not accept to give developers a $200/month subscription, even less for persona…

Calm down. I meant that my work covers my pro subscription.

Re: $500 GPU outperforms Claude Sonnet on coding benchmarks

#275

Post by 0xbadcafebee »

metalliqaz wrote: post #266
0xbadcafebee wrote: post #42

Earlier quoted context omitted.

No? You can run any model that fits in its VRAM, and you can run larger models with layer/MoE offloading. Ask an AI what the best models you can run on that card are, then ask it for newer models than that. Ask what tuning options to pass to llama.cpp, and what the auto-tuning options are. Use ROCm builds. It looks like your card has 16GB VRAM? Start with Qwen 3.5 9B Unsloth GGUFs (UD-Q6_K_XL) and branch out from the…

I've been running local models on my 9070XT and I have never found ROCm to be faster than Vulkan

It's not guaranteed to be faster, but it is faster in some cases due to ROCm taking more advantage of specific chipset features

Re: $500 GPU outperforms Claude Sonnet on coding benchmarks

#276

Post by senbrow »

bartread wrote: post #246
CraigJPerry wrote: post #170

Earlier quoted context omitted.

Do you not end up breaking half the value of referential integrity doing it that way (e.g. you had to update all the queries but now you have a sharp edge in that all future queries need to remember to be soft delete aware. Not a blocker for sure, just a sharp edge). You know your system better than me for sure, a random commenter on a website :-D your comment just shocked me out of my daze enough for my brain to say…

Yeah, I did consider moving records to shadow tables, but - because of the nature of our data - it requires moving a lot of child records as well, so it's quite a lot of additional churn in WAL, and the same for restore. And this approach has its own challenges with referential integrity. More than that, though: lots of queries for reporting, and the like, suddenly need to use JOINs. Same for admin use cases where we…

Thanks for taking the time to write such a high quality reply; this is something I've wondered about for a long time and I appreciate the thought and detail you've shared here. :)

Re: $500 GPU outperforms Claude Sonnet on coding benchmarks

#277

Post by slashdev »

imiric wrote: post #262
slashdev wrote: post #261

Earlier quoted context omitted.

You only hurt yourself with that attitude. AI might take your job.

> You only hurt yourself with that attitude. Funny, others seem more hurt by it. > AI might take your job. I'm not the one "grieving the loss of his career". :)

[dead]

Re: $500 GPU outperforms Claude Sonnet on coding benchmarks

#279

Post by XCSme »

BoorishBears wrote: post #270
XCSme wrote: post #95

Earlier quoted context omitted.

Why not? I described this in more detail in other comments. Even when using structured output, sometimes you want to define how the data should be displayed or formatted, especially for cases like chat bots, article writing, tool usage, calling external api's, parsing documents, etc. Most models get this right. Also, this is just one failure mode of Claude.

Like I said in the edit, when people want specific formatting they ask for well known formats: Markdown, XML, JSON I don't even need to debate if the benchmark is useful, it doesn't pass a sniff test: GPT-5.4 is not worse than Gemini 2.5 Flash in any way that matters to most users. In your benchmark it's meaningfully worse.

The questions do ask specifically to respond with the answer only, with an example format given in many cases.

Note that all reasoning models are tested with "medium" reasoning.

The benchmarks are questions/data processing tasks that an average user will likely ask, not coding questions (I didn't add any coding tests yet).

Gemini models also tend to be very consistent. Asking the same question will likely give the same result.

The two models you mention scored the same, the only difference is that Gemini was better at domain-specific questions (i.e. you ask something quite technical/niche).

Re: $500 GPU outperforms Claude Sonnet on coding benchmarks

#280

Post by XCSme »

scotty79 wrote: post #133
XCSme wrote: post #19

Earlier quoted context omitted.

Yup, they do quite poorly on random non-coding tasks: https://aibenchy.com/compare/minimax-minimax-m2-7-medium/moo...

GLM 5 here is significantly better than GPT-5.4

It's 8.3 vs 8.1, I wouldn't call that significantly better.

I think GLM got a bit in front, because on some tests that both got wrong, GLM did sometimes (inconsistently) respond with the correct answer.

That being said, yes, in this case probably with more and more tests added, gpt-5.4 would edge in front, especially if a coding would be added (there are no coding tests yet).

Post reply on HN