Not a word about the tok/sec, unfortunately.
> A single patched llama-server runs on K3s, providing both generation with speculative decoding (~100 tok/s)
There seems to be at least some detail on that point.
$500 GPU outperforms Claude Sonnet on coding benchmarks
31–40 of 311 posts
Re: $500 GPU outperforms Claude Sonnet on coding benchmarks
#32Earlier quoted context omitted.
> cheaper than the cost of local electricity only. Can you explain what that means?
I think they mean that the DeepSeek API charges are less than it would cost for the electricity to run a local model. Local model enthusiasts often assume that running locally is more energy efficient than running in a data center, but fail to take the economies of scale into account.
I guess it mostly comes from using the model with batch-size = 1 locally, vs high batch size in a DC, since GPU consumption don't grow that much with batch size.
Note that while a local chatbot user will mostly be using batch-size = 1, it's not going to be true if they are running an agentic framework, so the gap is going to narrow or even reverse.
Re: $500 GPU outperforms Claude Sonnet on coding benchmarks
#33It's a race to the bottom. DeepSeek beats all others (single-shot), and it is ~50% cheaper than the cost of local electricity only. > DeepSeek V3.2 Reasoning 86.2% ~$0.002 API, single-shot > ATLAS V3 (pass@1-v(k=3)) 74.6% ~$0.004 Local electricity only, best-of-3 + repair pipeline
You could use this approach with DeepSeek as well. The innovation here is that you can generate a bunch of solutions, use a small model to pick promising candidates and then test them. Then you feed errors back to the generator model and iterate. In a way, it's sort of like a genetic algorithm that converges on a solution.
Indeed but:
1) That is relatively very slow.
2) Can also be done, simpler even, with SoTA models over API.
Re: $500 GPU outperforms Claude Sonnet on coding benchmarks
#34[dead]
Re: $500 GPU outperforms Claude Sonnet on coding benchmarks
#35[flagged]
We've banned this account. Please don't post automated comments to HN.
Re: $500 GPU outperforms Claude Sonnet on coding benchmarks
#36I'm always skeptical because you can make it pass the benchmarks, then you use it and it is not practically useful unlike an extremely general model. Cool work though, really excited for the potential of slimming down models.
I find it's often very language and sector dependent. I still see a massive difference in systems programming (normally c++ and rust) between any open model I've tried and something like sonnet 4.5 (not really tried 4.6). And honestly, even the big models (like Opus 4.6) struggle in many cases.
Perhaps these things aren't well represented in the training data for these open models? Every local model I've tried (minimax2.5, GLM-4.7, Quen3, 3.5 and -coder variants) spend so much time trying to get something syntactically sensible and accepted by the compiler that when they've finished they barely seem to have any "momentum" left to actually solve the problems, as pretty much anything but the most trivial change ends up in another loop of actually trying to get it working again, often losing the intent of that change in the process.
My fear is that the solution here, having multiple instances all making the same changes for later comparison, would spend a huge amount of time beating it's head against compiler errors, types, memory allocation (NO DON'T JUST SPRINKLE IN A FEW MORE RAW "new" KEYWORDS DAMMIT) before it even gets to the "logic".
Having plenty of local GPU power I'd love to be able to actually use that, and I'm already wary about some of the training data use and it's interactions with the license of the code I'm "sending" to the cloud models...
Re: $500 GPU outperforms Claude Sonnet on coding benchmarks
#37I’d encourage devs to use MiniMax, Kimi, etc for real world tasks that require intelligence. The down sides emerge pretty fast: much higher reasoning token use, slower outputs, and degradation that is palpable. Sadly, you do get what you pay for right now. However that doesn’t prevent you from saving tons through smart model routing, being smart about reasoning budgets, and using max output tokens wisely. And optimiz…
I won’t use anything less than the SOTA. It tried using Opus 4.6 medium and immediately regretted it. High messes up enough.
Re: $500 GPU outperforms Claude Sonnet on coding benchmarks
#38I’d encourage devs to use MiniMax, Kimi, etc for real world tasks that require intelligence. The down sides emerge pretty fast: much higher reasoning token use, slower outputs, and degradation that is palpable. Sadly, you do get what you pay for right now. However that doesn’t prevent you from saving tons through smart model routing, being smart about reasoning budgets, and using max output tokens wisely. And optimiz…
Yup, they do quite poorly on random non-coding tasks: https://aibenchy.com/compare/minimax-minimax-m2-7-medium/moo...
It’s worth also comparing Qwen 3.5, it’s a very strong model. Different benchmarks give different results, but in general Qwen 3.5, GLM 5, and Kimi K2.5 are all excellent models, and not too far from current SOTA models in capability/intelligence. In my own non-coding tests, they were better than Gemini 3.1 flash. They’re comparable to the best American models from 6 months ago.
Re: $500 GPU outperforms Claude Sonnet on coding benchmarks
#39Earlier quoted context omitted.
You could use this approach with DeepSeek as well. The innovation here is that you can generate a bunch of solutions, use a small model to pick promising candidates and then test them. Then you feed errors back to the generator model and iterate. In a way, it's sort of like a genetic algorithm that converges on a solution.
Indeed but: 1) That is relatively very slow. 2) Can also be done, simpler even, with SoTA models over API.
Right, this works with any models. To me, the most interesting part is that you can use a smaller model that you could run locally to get results comparable to SoTA models. Ultimately, I'd far prefer running local, even if slower, for the simple reason of having sovereignty over my data.
Being reliant on a service means you have to share whatever you're working on with the service, and the service provider decides what you can do, and make changes to their terms of service on a whim.
If locally running models can get to the point where they can be used as a daily driver, that solves the problem.
Re: $500 GPU outperforms Claude Sonnet on coding benchmarks
#40This is specifically an experiment using ablation and multiple passes to improve the end result. Other techniques have been found that do this (like multiple passes through the same layers). But this technique - for this one specific model - seems to be both more performant, but also takes much longer, and requires more complexity. It's unlikely most people would use this technique, but it's interesting.