Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

$500 GPU outperforms Claude Sonnet on coding benchmarks

github.com

Post reply on HN
Print view View original

71–80 of 311 posts

Re: $500 GPU outperforms Claude Sonnet on coding benchmarks

#71

Post by nyargh »

hollerith wrote: post #60
bluefirebrand wrote: post #58

Earlier quoted context omitted.

I hope so too, but I think it's wishful thinking. Be prepared for the mother of all financial bailouts from the world governments to make sure that doesn't happen

I can understand why banks got bailed out by the US gov in 2008, but why would a government feel the need to bail out AI labs? I hope you are not going to say, "to avoid a global recession or depression caused by the popping of the AI bubble". That would be unnecessary and harmful (in its second-order effects), and governments do have advisors who are competent enough in economics to advise against such a move.

Bold of you to assume competency will overpower politics in our current era.

Re: $500 GPU outperforms Claude Sonnet on coding benchmarks

#72

Post by hollerith »

nyargh wrote: post #71
hollerith wrote: post #60

Earlier quoted context omitted.

I can understand why banks got bailed out by the US gov in 2008, but why would a government feel the need to bail out AI labs? I hope you are not going to say, "to avoid a global recession or depression caused by the popping of the AI bubble". That would be unnecessary and harmful (in its second-order effects), and governments do have advisors who are competent enough in economics to advise against such a move.

Bold of you to assume competency will overpower politics in our current era.

So far, the country I know best, the US, has been competent enough to avoid massive corporate bailouts except the aforementioned banks in 2008 and GM. The bailout of GM was not motivated by a desire to avoid a recession when a bubble pops.

If the AI labs become very influential and powerful, Washington might nationalize them, but that would be very different from bailing them out because they have become unprofitable and cannot attract additional investment from the private sector.

Re: $500 GPU outperforms Claude Sonnet on coding benchmarks

#73

Post by nyargh »

hollerith wrote: post #72
nyargh wrote: post #71

Earlier quoted context omitted.

Bold of you to assume competency will overpower politics in our current era.

So far, the country I know best, the US, has been competent enough to avoid massive corporate bailouts except the aforementioned banks in 2008 and GM. The bailout of GM was not motivated by a desire to avoid a recession when a bubble pops. If the AI labs become very influential and powerful, Washington might nationalize them, but that would be very different from bailing them out because they have become unprofitable…

Despite politics, TARP was arguably an economic success story for the US treasury despite public sentiment. Whether it created moral hazard or not I suppoae is up for debate.

GM on the other hand should have been left to die.

However, I was obliquely referring to the open transactionality and patronage encouraged by the current administration, and how the AI / big tech players have, with few exceptions, gleefully joined in.

Unless they run out of money for bribes, I think it's inevitable that current government will bend over backwards to prop them up.

Re: $500 GPU outperforms Claude Sonnet on coding benchmarks

#74

Post by freekh »

b3ing wrote: post #50

Will open source or local llms kill the big AI providers eventually? If so when? I can see maybe basic chat, not sure about coding and images yet

This has been my theory for a while: during this autumn Apple will release a version of Apple Intelligence that runs locally and works better than ChatGPT. They will do this because 1) they do not have an offering in AI yet 2) they have amazing hardware that even now almost can pull it off on open models and this will not be possible to replicate on android for a long time (presumably)

This will crush OpenAI.

Note: I am not talking about coding here - it will take a while longer but when it is optimized to the bone and llms output has stabilized, you will be running that too on local hardware. Cost will come down for Claude and friends too but why pay 5 when you can have it for free?

Re: $500 GPU outperforms Claude Sonnet on coding benchmarks

#75

Post by komali2 »

weird-eye-issue wrote: post #70
rf15 wrote: post #67

Earlier quoted context omitted.

You cannot afford the SOTA.

Why is that? The $200 per month subscription comes with a ton of usage. Opus 4.6 is available on the $20 plan too

I'm starting to think in these conversations we're all often talking about two different things. You're talking about running an LLM service through its provided tooling (codex, Claude, cursor), others seem to be talking token costs because they're integrating LLMs into software or are using harness systems like opencode, pi, or openclaw and balancing tasks across models.

Re: $500 GPU outperforms Claude Sonnet on coding benchmarks

#76

Post by bloppe »

Generating big chunks of code is rarely what I want from an agent. They really shine for stuff like combing through logs or scanning dozens of source files to explain a test failure. Which benchmark covers that? I want the debugging benchmark that tests mastery of build systems, CLIs, etc.

Re: $500 GPU outperforms Claude Sonnet on coding benchmarks

#77

Post by weird-eye-issue »

komali2 wrote: post #75
weird-eye-issue wrote: post #70

Earlier quoted context omitted.

Why is that? The $200 per month subscription comes with a ton of usage. Opus 4.6 is available on the $20 plan too

I'm starting to think in these conversations we're all often talking about two different things. You're talking about running an LLM service through its provided tooling (codex, Claude, cursor), others seem to be talking token costs because they're integrating LLMs into software or are using harness systems like opencode, pi, or openclaw and balancing tasks across models.

Fair enough, I read it quickly and assumed the person they replied to was talking about Claude Code

But I run a AI SaaS and we do offer Opus 4.6, too. Our use case is not nearly as token intensive as something like coding so we are still able to offer it with a good profit margin.

Also you can run OpenClaw with your CC subscription. It's what I do.

Re: $500 GPU outperforms Claude Sonnet on coding benchmarks

#79

Post by Leynos »

m00x wrote: post #68
mmaunder wrote: post #17

I’d encourage devs to use MiniMax, Kimi, etc for real world tasks that require intelligence. The down sides emerge pretty fast: much higher reasoning token use, slower outputs, and degradation that is palpable. Sadly, you do get what you pay for right now. However that doesn’t prevent you from saving tons through smart model routing, being smart about reasoning budgets, and using max output tokens wisely. And optimiz…

Minimax 2.7 is fine for most web stuff. It's slightly worse than Claude at backend, but works great for frontend. They're all slop when the complexity is higher than a mid-tech intermediate engineer though.

Kimi is surprisingly good at Rust.

Re: $500 GPU outperforms Claude Sonnet on coding benchmarks

#80

Post by rmi_ »

XCSme wrote: post #19
mmaunder wrote: post #17

I’d encourage devs to use MiniMax, Kimi, etc for real world tasks that require intelligence. The down sides emerge pretty fast: much higher reasoning token use, slower outputs, and degradation that is palpable. Sadly, you do get what you pay for right now. However that doesn’t prevent you from saving tons through smart model routing, being smart about reasoning budgets, and using max output tokens wisely. And optimiz…

Yup, they do quite poorly on random non-coding tasks: https://aibenchy.com/compare/minimax-minimax-m2-7-medium/moo...

Wild benchmark. Opus 4.6 is ranked #29, Gemini 3 Flash is #1, front of Pro.

I'm not saying it's bad, but it's definitely different than the others.

Post reply on HN