Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

$500 GPU outperforms Claude Sonnet on coding benchmarks

github.com

Post reply on HN
Print view View original

181–190 of 311 posts

Re: $500 GPU outperforms Claude Sonnet on coding benchmarks

#181

Post by TeMPOraL »

natebc wrote: post #168
TeMPOraL wrote: post #160

Earlier quoted context omitted.

Models aren't just big bags of floats you imagine them to be. Those bags are there, but there's a whole layer of runtimes, caches, timers, load balancers, classifiers/sanitizers, etc. around them, all of which have tunable parameters that affect the user-perceptible output.

There really always is a man behind the curtain eh?

It's still engineering. Even magic alien tech from outer space would end up with an interface layer to manage it :).

ETA: reminds me of biology, too. In life, it turns out the more simple some functional component looks like, the more stupidly overcomplicated it is if you look at it under microscope.

Re: $500 GPU outperforms Claude Sonnet on coding benchmarks

#182

Post by andyferris »

CraigJPerry wrote: post #170
bartread wrote: post #163

Earlier quoted context omitted.

I agree. Also good for small changes that need to be applied consistently across an entire codebase. I recently refactored our whole app from hard deletes to soft deletes. There are obviously various ways to skin this particular cat, but the way I chose needed all our deletions updated and also needed queries updating to exclude soft deleted rows, except in specific circumstances (e.g., admins restoring accidentally…

Do you not end up breaking half the value of referential integrity doing it that way (e.g. you had to update all the queries but now you have a sharp edge in that all future queries need to remember to be soft delete aware. Not a blocker for sure, just a sharp edge). You know your system better than me for sure, a random commenter on a website :-D your comment just shocked me out of my daze enough for my brain to say…

I move the record to another _index_, generally.

It depends whether you reliably control all the DB client code, of course.

Re: $500 GPU outperforms Claude Sonnet on coding benchmarks

#183

Post by mememememememo »

Epholys wrote: post #117
weird-eye-issue wrote: post #70

Earlier quoted context omitted.

Why is that? The $200 per month subscription comes with a ton of usage. Opus 4.6 is available on the $20 plan too

> The $200 per month subscription comes with a ton of usage. $200 dollars + VAT is half of my rent. I know HN is not a good place to rant on this subject, but I'm often flabbergasted about the number of people here that lives in a bubble with regard to the price of tech. Or just prices in general. I remember someone who said a few years ago (I'm paraphrasing): "You could just use one of the empty room in your house!"…

Thanks for the alternative perspective.

I think I am in the middle. I can afford $200/m but it'd be a brainer. And I don't pay that as I barely use home AI enough to warrant it.

I am also amazed at the richer end of HN but now I realize I am priviledged. Earned it? Like fuck I did. Lucky to be born a geek in late 20c. I'd be useless as a middle ages guy.

Re: $500 GPU outperforms Claude Sonnet on coding benchmarks

#184

Post by yorwba »

esskay wrote: post #116
hhh wrote: post #104

Earlier quoted context omitted.

The models don’t change.

Real world usage suggests otherwise. It's been a known trend for a while. Anthropic even confirmed as such ~6 months ago but said it was a "bug" - one that somehow just keeps happening 4-6 months after a model is released.

Real world usage is unlikely to give you the large sample sizes needed to reliably detect the differences between models. Standard error scales as the inverse square root of sample size, so even a difference as large as 10 percentage points would require hundreds of samples.

https://marginlab.ai/trackers/claude-code/ tries to track Claude Opus performance on SWE-Bench-Pro, but since they only sample 50 tasks per day, the confidence intervals are very wide. (This was submitted 2 months ago https://news.ycombinator.com/item?id=46810282 when they "detected" a statistically significant deviation, but that was because they used the first day's measurement as the baseline, so at some point they had enough samples to notice that this was significantly different from the long-term average. It seems like they have fixed this error by now.)

Re: $500 GPU outperforms Claude Sonnet on coding benchmarks

#185

Post by no_shadowban_3 »

sourcecodeplz wrote: post #48
selcuka wrote: post #10

It's a race to the bottom. DeepSeek beats all others (single-shot), and it is ~50% cheaper than the cost of local electricity only. > DeepSeek V3.2 Reasoning 86.2% ~$0.002 API, single-shot > ATLAS V3 (pass@1-v(k=3)) 74.6% ~$0.004 Local electricity only, best-of-3 + repair pipeline

I've tested many open models, Deepseek 3.2 is the only SOTA similar.

[dead]

Re: $500 GPU outperforms Claude Sonnet on coding benchmarks

#186

Post by vidarh »

mmaunder wrote: post #17

I’d encourage devs to use MiniMax, Kimi, etc for real world tasks that require intelligence. The down sides emerge pretty fast: much higher reasoning token use, slower outputs, and degradation that is palpable. Sadly, you do get what you pay for right now. However that doesn’t prevent you from saving tons through smart model routing, being smart about reasoning budgets, and using max output tokens wisely. And optimiz…

I get decent results with Kimi, but I agree with your overall premise. You do need to realise that while you can save money on a lot of tasks with those models, for the hardest tasks the "sticker price" of cost per million tokens isn't what matters.

It's also worth noting that the approach given in the link also benefits Sonnet and Opus. Not just as much - they are more forgiving - but put it in a harness that allows for various verification and repair and they too end up producing much better results than the "raw" model. And it's not clear that a harness around MiniMax, Kimi, or Qwen can measure up then.

I use those models a lot, and hope to use them more as my harnesses get better at discriminating which tasks they are cost effective for, but it's not straightforward to cost optimize this.

If I cared about running everything locally, then sure, it's amazing you can get to those kinds of results at all.

Re: $500 GPU outperforms Claude Sonnet on coding benchmarks

#187

Post by mememememememo »

Bombthecat wrote: post #124
Epholys wrote: post #117

Earlier quoted context omitted.

> The $200 per month subscription comes with a ton of usage. $200 dollars + VAT is half of my rent. I know HN is not a good place to rant on this subject, but I'm often flabbergasted about the number of people here that lives in a bubble with regard to the price of tech. Or just prices in general. I remember someone who said a few years ago (I'm paraphrasing): "You could just use one of the empty room in your house!"…

That's why ai is for the "rich". Poor people or later on middle class will be left behind....

Not sure. AI is sort of car ownership price. I think while that ain't poor, that is middle class.

So like if you want to start a business of any sort the AI sub is still peanuts.

AI is a car, or a dog, or a mild social life, or a utility bill level of cost. And thats for the level needed for a sane typical developer. (AI maximalists need 250k/y, let them slop it out)

It is not a Cessna, an infinity pool or a 1 month vacation.

Re: $500 GPU outperforms Claude Sonnet on coding benchmarks

#188

Post by rob0 »

Epholys wrote: post #152
hyperbovine wrote: post #140

Earlier quoted context omitted.

Work pays.

I'm not sure I've correctly understood what you're implying. If it's that I'm not working, well, I'm employed. It it's that I'm not working enough to not have this money... Well, we still go back to the bubble. Not everywhere in the world you can easily find a job that pays you enough, even if you accept to work more. And the employer will not accept to give developers a $200/month subscription, even less for persona…

> I could also, like someone said, put the tokens cost into my hourly/daily rate, but I'll be much more expensive than other freelancers.

Do you have any evidence of that? I think the OPs are assuming this as a premise so their logic is probably valid but may not be sound logic for you.

Re: $500 GPU outperforms Claude Sonnet on coding benchmarks

#189

Post by spiderfarmer »

Razengan wrote: post #154
spiderfarmer wrote: post #134

Earlier quoted context omitted.

"I don't get it. Everyone else is wrong."

"There's no such thing as astroturfing." ok I use Codex regularly and Claude is shit in comparison, from its constant "Oops you're right!!" backtracking to its crap Electron app (if their AI is so good why can't they make a fucking native app for each OS?) Hell right freakin now I asked it to implement something and got a weird "Something went wrong" API error

"Shit", "Crap", "Fucking", "Hell", "Freaking".

Maybe you're too easily frustrated. Or your existing code reads like your comments.

Re: $500 GPU outperforms Claude Sonnet on coding benchmarks

#190

Post by imtringued »

yogthos wrote: post #3
memothon wrote: post #2

I'm always skeptical because you can make it pass the benchmarks, then you use it and it is not practically useful unlike an extremely general model. Cool work though, really excited for the potential of slimming down models.

You obviously have to try it out to see how it works for you, but the trick they use is pretty clever. When you ask an AI to write code, it doesn’t always get it right. Sometimes the code has bugs, sometimes it misunderstands the problem entirely. A naive way to address that is to generate a few solutions and test each one. The odds that at least one works go way up. ATLAS generates multiple attempts, running each th…

I tried to read the project documentation, but I got overwhelmed by the aimless AI generated documentation that has a nebulous goal of documenting absolutely everything, but never explaining anything.

If the author actually wanted to explain his project he should have started with something along the lines of "Inference-time learning is the act of updating model parameters while you are generating tokens. Inference time learning is cost prohibitive for LLMs due to the need to update billions of parameters. However, what if updating billions of parameters wasn't necessary to begin with? What if you could instead have a much smaller model that merely scores a bunch of candidate output tokens? That model could be small enough for inference time learning to become viable and that's exactly what ATLAS does to achieve a 74.6% pass rate in LiveCodeBench and thereby outperforms Claude Sonnet with a small 14B open weight model that can be run locally on your $500 GPU."

This would have primed the reader to know what to look for. Instead you got this insurmountable wall of distractions.

Example: "combining constraint-driven generation, energy-based verification, self-verified iterative refinement, and adaptive routing"

That's a very long sequence of unexplained buzzwords that could mean absolutely anything.

Post reply on HN