Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Benchmarking Opus 5 on SlopCodeBench

github.com

Post reply on HN
Print view View original

111–120 of 131 posts

Re: Benchmarking Opus 5 on SlopCodeBench

#111

Post by svnt »

conception wrote: post #37
Espressosaurus wrote: post #29

Earlier quoted context omitted.

I’ve observed the degradation, but I suspect what’s happening is they’re tuning it for lower inference costs. Maybe turning down the amount of thinking, maybe quantizing, maybe something else. It seems like there’s a week by week and sometimes day by day change in performance when on a subscription plan using their harnesses.

https://marginlab.ai/trackers/claude-code/ their tracker generally shows that isn’t the case. The only times I’ve seen it drop is something broken and just before fable launched.

Is this using the api or using a subscription, though? The incentives are different for each, and it isn't the least bit unexpected that they would maintain API access quality while 'optimizing' the subscription experience to improve their margins (or losses)

It seems to do really this you would need to crowdsource it -- users individually give the lab access to a body of subscriptions normally used by average people, and the lab occasionally runs some masked version of the task through on diverse accounts.

Re: Benchmarking Opus 5 on SlopCodeBench

#112

Post by svnt »

xnorswap wrote: post #76
Gareth321 wrote: post #62

Earlier quoted context omitted.

I feel like a conspiracy theorist but it feels like every new model release from both Anthropic and OpenAI has 1-2 weeks of fantastic performance then a gradual (and sometimes not so gradual) decline in intelligence. It's like they're quantising the model in the background to optimise for available compute/RAM. Which, if I put my MBA hat on, would make perfect sense. Why not halve the required RAM for "90%" of the pe…

And yet: https://marginlab.ai/trackers/claude-code-historical-perform... There's clearly random variation, but it also shows each model release is just genuinely better. ( With the exception of a heavily degraded week of opus 4.7, which was acknowledged as a problem at the time. ) There's a psychology of getting used to models after being wowed by the new performance. It sets in as your new baseline expectations, and…

I replied to the user above that referenced marginlab, but I believe marginlab uses the API. It is possible (arguably likely, in MBA-land) that the API and subscription accounts hit different sub-models.

Even if they use a subscription account, surely Anthropic can tell which one it is.

Re: Benchmarking Opus 5 on SlopCodeBench

#113

Post by svnt »

ACCount37 wrote: post #78
Gareth321 wrote: post #62

Earlier quoted context omitted.

I feel like a conspiracy theorist but it feels like every new model release from both Anthropic and OpenAI has 1-2 weeks of fantastic performance then a gradual (and sometimes not so gradual) decline in intelligence. It's like they're quantising the model in the background to optimise for available compute/RAM. Which, if I put my MBA hat on, would make perfect sense. Why not halve the required RAM for "90%" of the pe…

You should. Feel like a conspiracy theorist when saying things like this. Users are not reliable or consistent model evaluators. Users adapt to models - the moment they get a model that performs better, their expectations rise, their tasks get harder and their prompts get shorter. "They made the model worse" is PEBKAC in 9 cases out of 10.

There are few conspiracies where the vectors between "capitalist organization makes more money" and "user can't reliably distinguish tiers of product quality" overlap.

Re: Benchmarking Opus 5 on SlopCodeBench

#114

Post by tracker1 »

WilcoKruijer wrote: post #67

I hope the big labs will start using this benchmark in their RL pipelines. Reducing complexity in generated code should be the number 1 priority, in my opinion. The holy grail for me is models implementing features while reducing LoC (i.e., choosing the right abstractions). What is also nice about this benchmark is that it can be used to iterate on prompts/skills for reducing code complexity.

I call out specifically in my agent/claude file that I want and prefer simpler solutions over complex enterprise pattern usage and overuse of abstractions. It tends to help a bit, but often I have to give feedback in my review step.

My typical workflow when AI assisted is what I call human gatekeeping... I'll plan out next steps with the agent, updating a TODO.md file with what needs to be done, then a fresh context to implement the next step(s), and review the code before committing to git. I may iterate/stash or even reset a few times before it's "good enough"... it is rarely close to what I would do myself, but often as good as what I've gotten from other developers IRL on projects.

This includes updating the documentation area(s) of a project as well as testing. I also tend to lean on ./run/* for scripts that will run/test various portions of the project... getting the agent to use these is sometimes harder than it should be, as I want it to specifically exercise a lot of things through the process... there are also times where it will try to change a valid test that's failing instead of fixing the code. That's the most irritating part. Some models are annoying, some feel like pure magic at times.

Re: Benchmarking Opus 5 on SlopCodeBench

#115

Post by fastball »

conception wrote: post #36
fastball wrote: post #21

Earlier quoted context omitted.

Medium vs High? Why? From all the charts I've seen the performance jump is pretty large from med -> high (not as noticeable from high -> xhigh).

https://cognition.com/frontiercode Quality vs cost - medium is the sweet (perhaps better too!) spot.

That is just a single benchmark tho

Re: Benchmarking Opus 5 on SlopCodeBench

#116

Post by 8note »

Espressosaurus wrote: post #29
swader999 wrote: post #28

Earlier quoted context omitted.

Noticed that too. I wonder if these things just degrade over time, perhaps with the way it writes memories about my project as it goes

I’ve observed the degradation, but I suspect what’s happening is they’re tuning it for lower inference costs. Maybe turning down the amount of thinking, maybe quantizing, maybe something else. It seems like there’s a week by week and sometimes day by day change in performance when on a subscription plan using their harnesses.

i thought i had noticed a degradation, but it turned out claude code had swapped itself back to opus.

might be the case for you as well

Re: Benchmarking Opus 5 on SlopCodeBench

#118

Post by jefffoster »

Vgoose wrote: post #17

Nice! I actually ran across this paper+benchmark recently, too. It's the first I've found that start to aim at some of the non-functional and longitudinal requirements that I think have always been an important part of writing production code. It's especially relevant now that models are good enough to solve ~most point-in-time problems. Some relevant but disconnected thoughts: - deterministic scores are so nice - wh…

I’ve been thinking about this too.

There’s this notion of variety popularized by cybernetics folks a long time ago. Variety is like the state space of the system. Then there’s a law that says “only variety absorbs variety”.

So if a method has high variety then it must have an equally complex implementation to handle the variety.

When there is a mismatch it means that either the method has parameters that aren’t useful, or that the body of the method isn’t covering cases it should.

https://fffej.substack.com/p/only-variety-can-absorb-variety was my attempt to write it up more fully.

Re: Benchmarking Opus 5 on SlopCodeBench

#119

Post by robbomacrae »

dhorthy wrote: post #108
robbomacrae wrote: post #65

SCB is definitely an underrated benchmark. For me the unique selling point is that it more closely mirrors software development by not stopping after a single task. The agent has to keep code clean. The only disadvantage is all the problems are greenfield and not git inited so the agents don’t make use of git diffs. I’ve used SCB as part of my assessment of agent skills (superpowers, GSD etc) https://orcabot.com/labs…

so wait is the finding that most of those skills reduce pass rates against SCB? wild

Yep that was interesting but imho not completely unexpected. My original hypothesis was that these agent skills are castles built on sand and that the models would quickly adapt and make them redundant. Also the skills use up context and I think a lot of them partially focus on investigating a large complex codebase properly... which is wasted on greenfield projects.
Post reply on HN