Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

$500 GPU outperforms Claude Sonnet on coding benchmarks

github.com

Post reply on HN
Print view View original

211–220 of 311 posts

Re: $500 GPU outperforms Claude Sonnet on coding benchmarks

#211

Post by MattRix »

memothon wrote: post #2

I'm always skeptical because you can make it pass the benchmarks, then you use it and it is not practically useful unlike an extremely general model. Cool work though, really excited for the potential of slimming down models.

I think this is because when you shrink it down, the model ends up space constrained and each “neuron” ends up having to do multiple duties. It can stil be tuned to perform well at specific tasks, but no longer generalizes as well. It’s somewhat unintuitive but models that are larger are often simpler than smaller ones for this same reason.

Re: $500 GPU outperforms Claude Sonnet on coding benchmarks

#212

Post by seunosewa »

bloppe wrote: post #76

Generating big chunks of code is rarely what I want from an agent. They really shine for stuff like combing through logs or scanning dozens of source files to explain a test failure. Which benchmark covers that? I want the debugging benchmark that tests mastery of build systems, CLIs, etc.

Create it!

Re: $500 GPU outperforms Claude Sonnet on coding benchmarks

#213

Post by naasking »

zar1048576 wrote: post #4
yogthos wrote: post #3

Earlier quoted context omitted.

You obviously have to try it out to see how it works for you, but the trick they use is pretty clever. When you ask an AI to write code, it doesn’t always get it right. Sometimes the code has bugs, sometimes it misunderstands the problem entirely. A naive way to address that is to generate a few solutions and test each one. The odds that at least one works go way up. ATLAS generates multiple attempts, running each th…

Really intriguing set of techniques to improve accuracy by generating multiple solutions. Even with the work to predict the most likely solutions, it's not clear to me based on the description how this could all be done efficiently. Would definitely be really impressive if it pans out on real-world use cases. Will look to kick the tires on this if I can get some time.

> it's not clear to me based on the description how this could all be done efficiently.

Depends how you define efficiency. The power use of this rig is a lot less than the large data centers that serve trillion parameter models. The page suggests that the final dollar cost per request is an order of magnitude lower than the frontier models charge.

Re: $500 GPU outperforms Claude Sonnet on coding benchmarks

#214

Post by slashdev »

bloppe wrote: post #76

Generating big chunks of code is rarely what I want from an agent. They really shine for stuff like combing through logs or scanning dozens of source files to explain a test failure. Which benchmark covers that? I want the debugging benchmark that tests mastery of build systems, CLIs, etc.

Generating big chunks code is all I do, all day.

I don't write code by hand any more, neither at work, nor for side projects.

I work mostly in Rust and TypeScript at a developer tools company.

Re: $500 GPU outperforms Claude Sonnet on coding benchmarks

#215

Post by imiric »

slashdev wrote: post #214
bloppe wrote: post #76

Generating big chunks of code is rarely what I want from an agent. They really shine for stuff like combing through logs or scanning dozens of source files to explain a test failure. Which benchmark covers that? I want the debugging benchmark that tests mastery of build systems, CLIs, etc.

Generating big chunks code is all I do, all day. I don't write code by hand any more, neither at work, nor for side projects. I work mostly in Rust and TypeScript at a developer tools company.

[flagged]

Re: $500 GPU outperforms Claude Sonnet on coding benchmarks

#216

Post by littlestymaar »

eru wrote: post #97
littlestymaar wrote: post #32

Earlier quoted context omitted.

I guess it mostly comes from using the model with batch-size = 1 locally, vs high batch size in a DC, since GPU consumption don't grow that much with batch size. Note that while a local chatbot user will mostly be using batch-size = 1, it's not going to be true if they are running an agentic framework, so the gap is going to narrow or even reverse.

Well, different parts of the world also have different electricity prices.

Usually not multiple orders of magnitude difference though.

Re: $500 GPU outperforms Claude Sonnet on coding benchmarks

#217

Post by yogthos »

eru wrote: post #96
yogthos wrote: post #25

Earlier quoted context omitted.

You could use this approach with DeepSeek as well. The innovation here is that you can generate a bunch of solutions, use a small model to pick promising candidates and then test them. Then you feed errors back to the generator model and iterate. In a way, it's sort of like a genetic algorithm that converges on a solution.

Why do you need a small model to pick promising candidates? Why not a bigger one? (And ideally you'd probably test first, or at least try to feed compiler errors back etc?) Overall, I mostly agree.

mostly an issue of speed and resource usage, if the model is too big then simply running the tests will be cheaper

Re: $500 GPU outperforms Claude Sonnet on coding benchmarks

#218

Post by josefritzishere »

The core problem of AI remains unresolved, with no conceivable path to solvency. The issue is that AI isn't very good. It's OK, sometimes under very narrow criteria. But providing AI in reality very costly. Vague promises of it magically becoming better remain, very optimistic at best and still provide no route to solvency.

Re: $500 GPU outperforms Claude Sonnet on coding benchmarks

#219

Post by aditmag »

imiric wrote: post #215
slashdev wrote: post #214

Earlier quoted context omitted.

Generating big chunks code is all I do, all day. I don't write code by hand any more, neither at work, nor for side projects. I work mostly in Rust and TypeScript at a developer tools company.

[flagged]

Tbf, as long as you really know what you're doing and have the sense to avoid falling into a spaghetti code trap, generating bigger chunks of code absolutely works and should be done. The pitfall happens when

(a) the dev has no idea what the agent is doing (b) the dev gives overtly-broad instructions.

If you give it specific enough tasks (not to the point where it's writing singular functions) but a general class description, you're on a good track.

Re: $500 GPU outperforms Claude Sonnet on coding benchmarks

#220

Post by kroaton »

mongrelion wrote: post #171
sznio wrote: post #105

On that topic, anyone here got a decent local coding AI setup for a 12GB VRAM system? I have a Radeon 6700 XT and would like to run autocomplete on it. I can fit some models in the memory and they run quick but are just a tad too dumb. I have 64GB of system ram so I can run larger models and they are at least coherent, but really slow compared to running from VRAM.

Not the answer that you are looking for, but I am a fellow AMD GPU owner, so I want to share my experience. I have a 9070 XT, which has 16GB of VRAM. My understanding from reading around a bunch of forums is that the smallest quant you want to go with is Q4. Below that, the compression starts hurting the results quite a lot, especially for agentic coding. The model might eventually start missing brackets, quotes, etc…

For autocomplete, Qwen 3.5 9B should be enough even at Q4_k_m. The upcoming coding/math Omnicoder-2 finetune might be useful (should be released in a few days).

Either that or just load up Qwen3.5-35B-A3B-Q4_K_S I'm serving it at about 40-50t/s on a 4070RTX Super 12GB + 64GB of RAM. The weights are 20.7GB + KV Cache (which should be lowered soon with the upcoming addition of TurboQuant).

Post reply on HN