Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

$500 GPU outperforms Claude Sonnet on coding benchmarks

github.com

Post reply on HN
Print view View original

121–130 of 311 posts

Re: $500 GPU outperforms Claude Sonnet on coding benchmarks

#121

Post by stuaxo »

dvt wrote: post #69
m00x wrote: post #68

Earlier quoted context omitted.

Minimax 2.7 is fine for most web stuff. It's slightly worse than Claude at backend, but works great for frontend. They're all slop when the complexity is higher than a mid-tech intermediate engineer though.

> They're all slop when the complexity is higher than a mid-tech intermediate engineer though. This right here. Value prop quickly goes out the window when you're building anything novel or hard. I feel that I'm still spending the same amount of time working on stuff, except that now I'm also spending money on models.

10x more code output is 10x more review.

We've gone from doing the first 90% and then the second 90% to the first 90% and the second 990%, its exausting.

Re: $500 GPU outperforms Claude Sonnet on coding benchmarks

#122

Post by fer »

hhh wrote: post #104
withinboredom wrote: post #66

Earlier quoted context omitted.

Opus 4.5 ~= Opus 4.6 high. Opus 4.5 was nerfed just before or after the release of 4.6.

The models don’t change.

They do. I'm currently seeing a degradation on Opus 4.6 on tasks it could do without trouble a few months back. Obvious I'm a sample of n=1, but I'm also convinced a new model is around the corner and they preemptively nerf their current model so people notice the "improvement".

Re: $500 GPU outperforms Claude Sonnet on coding benchmarks

#123

Post by Bombthecat »

bloppe wrote: post #76

Generating big chunks of code is rarely what I want from an agent. They really shine for stuff like combing through logs or scanning dozens of source files to explain a test failure. Which benchmark covers that? I want the debugging benchmark that tests mastery of build systems, CLIs, etc.

Oh yes! I let my environments now be built by agents via kubectl / helm and let them debug issues.

It's amazing! Saves hours of work!

I create the basic helm configd settings etc and when there is a conflict or something not working I let an agent fix it!

Re: $500 GPU outperforms Claude Sonnet on coding benchmarks

#124

Post by Bombthecat »

Epholys wrote: post #117
weird-eye-issue wrote: post #70

Earlier quoted context omitted.

Why is that? The $200 per month subscription comes with a ton of usage. Opus 4.6 is available on the $20 plan too

> The $200 per month subscription comes with a ton of usage. $200 dollars + VAT is half of my rent. I know HN is not a good place to rant on this subject, but I'm often flabbergasted about the number of people here that lives in a bubble with regard to the price of tech. Or just prices in general. I remember someone who said a few years ago (I'm paraphrasing): "You could just use one of the empty room in your house!"…

That's why ai is for the "rich". Poor people or later on middle class will be left behind....

Re: $500 GPU outperforms Claude Sonnet on coding benchmarks

#125

Post by LoganDark »

aleph_minus_one wrote: post #113
weird-eye-issue wrote: post #70

Earlier quoted context omitted.

Why is that? The $200 per month subscription comes with a ton of usage. Opus 4.6 is available on the $20 plan too

> The $200 per month subscription comes with a ton of usage. 200 USD/month is a number only really affluent programmers (e.g. in the Silicon Valley) can perhaps pay easily.

> 200 USD/month is a number only really affluent programmers (e.g. in the Silicon Valley) can perhaps pay easily.

Not true, I live in USA PNW and my last remote job paid $12k/mo. I have been jobless for over a month now (currently waiting for the next HN "who wants to be hired"), but I still have enough savings to easily afford to continue that plan for a while.

I don't think it really has to do with affluence but more the job market and economy you're in. Countries with lower salaries or higher costs of living will have less buying power.

Re: $500 GPU outperforms Claude Sonnet on coding benchmarks

#126

Post by raincole »

XCSme wrote: post #19
mmaunder wrote: post #17

I’d encourage devs to use MiniMax, Kimi, etc for real world tasks that require intelligence. The down sides emerge pretty fast: much higher reasoning token use, slower outputs, and degradation that is palpable. Sadly, you do get what you pay for right now. However that doesn’t prevent you from saving tons through smart model routing, being smart about reasoning budgets, and using max output tokens wisely. And optimiz…

Yup, they do quite poorly on random non-coding tasks: https://aibenchy.com/compare/minimax-minimax-m2-7-medium/moo...

I can't imagine anyone looking at this benchmark without laughing. It's so disconnected.

Re: $500 GPU outperforms Claude Sonnet on coding benchmarks

#127

Post by [deleted] »

Epholys wrote: post #117
weird-eye-issue wrote: post #70

Earlier quoted context omitted.

Why is that? The $200 per month subscription comes with a ton of usage. Opus 4.6 is available on the $20 plan too

> The $200 per month subscription comes with a ton of usage. $200 dollars + VAT is half of my rent. I know HN is not a good place to rant on this subject, but I'm often flabbergasted about the number of people here that lives in a bubble with regard to the price of tech. Or just prices in general. I remember someone who said a few years ago (I'm paraphrasing): "You could just use one of the empty room in your house!"…

[deleted]

Re: $500 GPU outperforms Claude Sonnet on coding benchmarks

#128

Post by cpursley »

aleph_minus_one wrote: post #113
weird-eye-issue wrote: post #70

Earlier quoted context omitted.

Why is that? The $200 per month subscription comes with a ton of usage. Opus 4.6 is available on the $20 plan too

> The $200 per month subscription comes with a ton of usage. 200 USD/month is a number only really affluent programmers (e.g. in the Silicon Valley) can perhaps pay easily.

Are you kidding me? Even developer salaries in the Philippines can afford that or at least the plan below it. If I used the Anthropic API, my monthly spend would be $4k a month. The Claude Max plan is the best bargain around.

Re: $500 GPU outperforms Claude Sonnet on coding benchmarks

#129

Post by walletdrainer »

Epholys wrote: post #117
weird-eye-issue wrote: post #70

Earlier quoted context omitted.

Why is that? The $200 per month subscription comes with a ton of usage. Opus 4.6 is available on the $20 plan too

> The $200 per month subscription comes with a ton of usage. $200 dollars + VAT is half of my rent. I know HN is not a good place to rant on this subject, but I'm often flabbergasted about the number of people here that lives in a bubble with regard to the price of tech. Or just prices in general. I remember someone who said a few years ago (I'm paraphrasing): "You could just use one of the empty room in your house!"…

>I'm often flabbergasted about the number of people here that lives in a bubble with regard to the price of tech

Sorry, no. You live in the bubble, the people you think are living in a bubble are actually doing the very opposite and taking advantage of the lack of bubbles in our globally connected world.

Today, basically anyone can sell any bullshit to billions of people around the world. We’ve never lived in less of a bubble.

Re: $500 GPU outperforms Claude Sonnet on coding benchmarks

#130

Post by comboy »

XCSme wrote: post #90
comboy wrote: post #89

Earlier quoted context omitted.

Not really related, but does anybody know if somebody's tracking same models performance on some benchmarks over time? Sometimes I feel like I'm being A/B tested.

Oh, I didn't think about this, that's a good idea. I also feel generally model performance changes over time (usually it gets worse). The problem with doing this is cost. Constsntly testing a lot of models on a large dataset can get really costly.

Yeah, good tests are associated with cost. I'd like to see benchmarks on big messy codebases and how models perform on a clearly defined task that's easy to verify.

I was thinking that tokens spent in such case could also be an interesting measure, but some agent can do small useful refactoring. Although prompt could specify to do the minimal change required to achieve the goal.

Post reply on HN