Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

$500 GPU outperforms Claude Sonnet on coding benchmarks

github.com

Post reply on HN
Print view View original

201–210 of 311 posts

Re: $500 GPU outperforms Claude Sonnet on coding benchmarks

#201

Post by throwaway173738 »

mememememememo wrote: post #183
Epholys wrote: post #117

Earlier quoted context omitted.

> The $200 per month subscription comes with a ton of usage. $200 dollars + VAT is half of my rent. I know HN is not a good place to rant on this subject, but I'm often flabbergasted about the number of people here that lives in a bubble with regard to the price of tech. Or just prices in general. I remember someone who said a few years ago (I'm paraphrasing): "You could just use one of the empty room in your house!"…

Thanks for the alternative perspective. I think I am in the middle. I can afford $200/m but it'd be a brainer. And I don't pay that as I barely use home AI enough to warrant it. I am also amazed at the richer end of HN but now I realize I am priviledged. Earned it? Like fuck I did. Lucky to be born a geek in late 20c. I'd be useless as a middle ages guy.

If I found myself in the middle ages I’d just become a blacksmith or a miller.

Re: $500 GPU outperforms Claude Sonnet on coding benchmarks

#202

Post by freekh »

oarsinsync wrote: post #100
freekh wrote: post #74

Earlier quoted context omitted.

This has been my theory for a while: during this autumn Apple will release a version of Apple Intelligence that runs locally and works better than ChatGPT. They will do this because 1) they do not have an offering in AI yet 2) they have amazing hardware that even now almost can pull it off on open models and this will not be possible to replicate on android for a long time (presumably) This will crush OpenAI. Note: I…

> This has been my theory for a while: during this autumn Apple will release a version of Apple Intelligence that runs locally and works better than ChatGPT. In this theory, can you explain why Apple has announced it’s paying Google for Gemini too? Eventually, this may be true. This autumn? Highly unlikely.

The Google Gemini deal is one of the reasons I think it is likely since Gemini works pretty local hw...

Re: $500 GPU outperforms Claude Sonnet on coding benchmarks

#203

Post by Epholys »

weird-eye-issue wrote: post #177
Epholys wrote: post #158

Earlier quoted context omitted.

First, I've assumed you were in the bubble I described, but that's not the case, so sorry bout that. Also, I think it's relevant to the conversation. You replied to someone who said that "you" (undirected pronoun I suppose) can't afford the SOTA that the $200/month Anthropic subscription comes with a ton of usage. So I interpreted it as a general statement. It wasn't what you meant? I'm a bit lost about who you're ta…

I assume when somebody says you and is not talking about anyone in particular they mean that it's infeasible for virtually everybody which is certainly not the case. Also you conveniently disregarded the fact that is available on the $20 per month plan.

Okay, I understand better. I interpreted your answer as "well, it's $200, everybody can afford it". Clearly a misunderstanding.

Going back to the $20 plan, yes, I agree it's much more accessible.

I didn't talk about it because I've seen a lot of comments here, on blogs, on social media about how a $200 subscription for Claude is a no brainer. And it got on my nerves, so I wanted to tell how much money it can be. To you (and it was misguided reading your answers), and to concerned HN commenters in general.

Re: $500 GPU outperforms Claude Sonnet on coding benchmarks

#204

Post by jacquesm »

jLaForest wrote: post #49
jacquesm wrote: post #43

Earlier quoted context omitted.

Some of those local model enthusiasts can actually afford solar panels.

You are still incurring a cost if you use the electricity instead of selling it back to the grid

Selling it back to the grid is something that is still possible but much, much less of a financially sound proposition than it was a few years ago because of regulatory capture by the utilities. In some places it is so bad that you get penalized for excess power. Local consumption is the fastest way to capitalize on this, more so if you can make money with that excess power.

Re: $500 GPU outperforms Claude Sonnet on coding benchmarks

#205

Post by Epholys »

coldtea wrote: post #197
Epholys wrote: post #117

Earlier quoted context omitted.

> The $200 per month subscription comes with a ton of usage. $200 dollars + VAT is half of my rent. I know HN is not a good place to rant on this subject, but I'm often flabbergasted about the number of people here that lives in a bubble with regard to the price of tech. Or just prices in general. I remember someone who said a few years ago (I'm paraphrasing): "You could just use one of the empty room in your house!"…

In the US/Western Europe? Because for devs especially in the former, $200 is pocket change, especially for a core productivity tool. And the rent would be in the $1200 to $3000 easily. Same for houses. Maybe not in NY or SF, but in most of the US there's no shortage of house spaces and redundant rooms.

I've seen those comments about $200/month and empty rooms here, so I suppose they mainly come from the US, yes.

So yes, you describe a situation that I feel like a lot of people here don't understand is not the norm.

I compared the subscription with my rent precisely because it's easier to compare: with your numbers it would be like paying from $600 up to $1500 / month. Pretty hard to justify.

Re: $500 GPU outperforms Claude Sonnet on coding benchmarks

#206

Post by withinboredom »

nl wrote: post #180
woadwarrior01 wrote: post #173

Earlier quoted context omitted.

There's this[1]. Model providers have a strong incentive to switch (a part of) their inference fleet to quantized models during peak loads. From a systems perspective, it's just another lever. Better to have slightly nerfed models than complete downtime. [1]: https://marginlab.ai/trackers/claude-code/

So - as the charts say - no statistical difference? Isn't this link am argument against the point you are making?

The chart doesn't cover the 4.6 release which was in the end of December/early January time frame. So, it's hard to tell from existing data.

Re: $500 GPU outperforms Claude Sonnet on coding benchmarks

#207

Post by naasking »

xyzzy123 wrote: post #13
yogthos wrote: post #6

Earlier quoted context omitted.

Seems like the key insight is to train a small model that acts as a heuristic for embeddings that resemble quality code. I imagine a lot depends on how well this model is trained. And you could probably create specialized versions for different languages and domains. Another interesting approach could be to use this set up with a language like Clojure or Common Lisp which facilitates interactive development. If you c…

I'm super confused. The small model "cost field" `rag-api/geometric_lens/cost_field.py` was trained on PASS_TASKS like "Write a function that counts vowels in a string." and FAIL_TASKS like "Write a function that converts a regular expression string to an NFA using Thompson's construction, then converts the NFA to a DFA.". So it seems like it's a difficulty classifier for task descriptions written in English. This is…

> But none of this helps you solve harder problems, or distinguish between a simple solution which is wrong, and a more complex solution which is correct.

It does because hallucinations and low confidence share characteristics in the embedding vector which the small neural learns to recognize. And the fact that it continuously learns based on the feedback loop is pretty slick.

Re: $500 GPU outperforms Claude Sonnet on coding benchmarks

#208

Post by weird-eye-issue »

m4rtink wrote: post #131
weird-eye-issue wrote: post #70

Earlier quoted context omitted.

Why is that? The $200 per month subscription comes with a ton of usage. Opus 4.6 is available on the $20 plan too

A subscription for coding - no thanks.

If you think it's only for coding you don't have much of an imagination :)

Re: $500 GPU outperforms Claude Sonnet on coding benchmarks

#209

Post by Epholys »

rob0 wrote: post #188
Epholys wrote: post #152

Earlier quoted context omitted.

I'm not sure I've correctly understood what you're implying. If it's that I'm not working, well, I'm employed. It it's that I'm not working enough to not have this money... Well, we still go back to the bubble. Not everywhere in the world you can easily find a job that pays you enough, even if you accept to work more. And the employer will not accept to give developers a $200/month subscription, even less for persona…

> I could also, like someone said, put the tokens cost into my hourly/daily rate, but I'll be much more expensive than other freelancers. Do you have any evidence of that? I think the OPs are assuming this as a premise so their logic is probably valid but may not be sound logic for you.

I don't have any hard evidence, no.

Instinctively, if we suppose all the newbies freelancers without any reputation start with the same lowest rate possible to be competitive, passing additional cost to my client will mechanically increase my rate. Putting me in disadvantage about getting any work. And with the difference of monetary value for the same price of tokens, the rate delta is higher.

It's a simplified model of the world, but it feels like simple economic rules.

I assume the comment I'm referring to was written by someone who is already established and for Wich the token cost passing is lower relatively to my environment.

Re: $500 GPU outperforms Claude Sonnet on coding benchmarks

#210

Post by yorwba »

nextaccountic wrote: post #200
yorwba wrote: post #184

Earlier quoted context omitted.

Real world usage is unlikely to give you the large sample sizes needed to reliably detect the differences between models. Standard error scales as the inverse square root of sample size, so even a difference as large as 10 percentage points would require hundreds of samples. https://marginlab.ai/trackers/claude-code/ tries to track Claude Opus performance on SWE-Bench-Pro, but since they only sample 50 tasks per day,…

It's hard to trust public, high profile benchmarks because any change to a specific model (Opus 4.5 in this case) can be rejected if they have regressions on SWE-Bench-Pro, so everything that gets to be released would perform well in this benchmark

Any other benchmark at that sample size would have similarly huge error bars. Unless Anthropic makes a model that works 100% of the time or writes a bug that brings it all the way to zero, it's going to work sometimes and fail sometimes, and anyone who thinks they can spot small changes in how often it works without running an astonishingly large number of tests is fooling themselves with measurement noise.
Post reply on HN