Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Benchmarking Opus 5 on SlopCodeBench

github.com

Post reply on HN
Print view View original

41–50 of 131 posts

Re: Benchmarking Opus 5 on SlopCodeBench

#41

Post by killingtime74 »

dhorthy wrote: post #9
killingtime74 wrote: post #7

Did you not benchmark latest GPT 5.6 or GLM 5.1/Kimi K3 because of cost? I can run them if you share how you ran them

no i'm spinning those up at some point this week. here's the first few prompts I used (claude opus 5 as the research orchestrator), (these were interspersed with lots of tools and assistant messages but it should get you kicked off. > fetch this article for slopcodebench and help me run an eval on a subset of problems with opus 5 https://arxiv.org/html/2603.24755v1 > Get all the context, fetch any mentioned repos, an…

Thank you!

Re: Benchmarking Opus 5 on SlopCodeBench

#42

Post by ValentineC »

fastball wrote: post #21
sothatsit wrote: post #18

This matches my experience of Opus 5 being a nice improvement over Opus 4.8, but not being revolutionary like Fable felt. I’ve now replaced my use of Opus 4.8 xhigh with Opus 5 medium, and I’m using less tokens and it’s quicker. I can understand people being annoyed by its writing style but for getting work done that really doesn’t bother me. I’ve been really enjoying using it.

Medium vs High? Why? From all the charts I've seen the performance jump is pretty large from med -> high (not as noticeable from high -> xhigh).

Medium or low supposedly prevents Opus 5 from overthinking:

https://xcancel.com/danshipper/status/2080700057892815114

Re: Benchmarking Opus 5 on SlopCodeBench

#44

Post by lostdog »

ajwin wrote: post #23

To what degree is this a harness/system prompt problem? Models maybe should implement new stuff with as little impact on the existing stuff as possible by default? A simple system prompt for it to always check the code after task completion for proper simplifications, abstractions and cleanups before returning to the user? Instructions to retain "story like" readability of the code.

Clearly the first step of slopbench2 should be to have the agent first write its own harness!

Re: Benchmarking Opus 5 on SlopCodeBench

#45

Post by dhorthy »

cesarvarela wrote: post #43

Please add Fable; a good benchmark should show that Fable is less prone to just autocomplete and instead pushes back or is at least more tasteful.

Yeah the main reason I skipped fable was because we have a ZDR with anthropic and I didn’t feel like spinning up another account to circumvent that. Next run will have fable and sol

Re: Benchmarking Opus 5 on SlopCodeBench

#46

Post by dhorthy »

igregoryca wrote: post #26
ajwin wrote: post #23

To what degree is this a harness/system prompt problem? Models maybe should implement new stuff with as little impact on the existing stuff as possible by default? A simple system prompt for it to always check the code after task completion for proper simplifications, abstractions and cleanups before returning to the user? Instructions to retain "story like" readability of the code.

At least for Claude Code, putting "run /simplify at the end" in an "implement the plan" skill helps a little. It still often leaves new code in bizarre places, and/or with bad/alien-sounding names and comments.

Yeah I would hold that models don’t know how to simplify because most rl/benchmarks doesn’t penalize complexity

Re: Benchmarking Opus 5 on SlopCodeBench

#48

Post by 4by4by4 »

I'd be curious to see the raw test results.

I have a suspicion that most models will miss the `database_migration` Checkpoint 2 test that includes a `default_value` because it could be interpreted as either a JSON-literal or a SQL-expression.

There might be other tests as well that are prone to failure for reasons other than the reasons cited in the paper.

I think a cool experiment would be to adjust the order of the features implemented (e.g. checkpoint 3 then 2 then 5 then 4) where dependencies allow it. Then one could account for some checkpoints being more difficult than others.

Re: Benchmarking Opus 5 on SlopCodeBench

#49

Post by latentsea »

patwolf wrote: post #34
cute_boi wrote: post #12

Opus 5 is an overconfident stupid model. It tries to generate too much slop, tries to act like everything will fall. I have reversed back to fable and codex sol.

We've been running automated code reviews on claude with a bunch of skills/subagents with different specialties. Any review feedback is then fed back into claude to fix. Since switching to Opus 5 I've noticed the reviews are overly pedantic, and that leads to feedback loops where each fix generates more feedback, which requires more fixes, i.e. slop. I had, for example, a simple SQL migration script with a single CRE…

> I'm not ready to blame Opus 5 for being stupid. Perhaps we have a prompt buried somewhere that's essentially asking it to be pedantic, and it's just obeying the prompt.

This is a known failure mode. Sadly, working on a software engineering team doing agentic engineering now means we need to build and maintain suites of evals that measure these things, so that we can measure the effects of changes to harness components including how they perform under model upgrades.

But... for a traditional software engineering team that has no experience in this... How do we even do it?

Re: Benchmarking Opus 5 on SlopCodeBench

#50

Post by dhorthy »

4by4by4 wrote: post #48

I'd be curious to see the raw test results. I have a suspicion that most models will miss the `database_migration` Checkpoint 2 test that includes a `default_value` because it could be interpreted as either a JSON-literal or a SQL-expression. There might be other tests as well that are prone to failure for reasons other than the reasons cited in the paper. I think a cool experiment would be to adjust the order of the…

oh i really like the idea of flipping around the order of checkpoints and comparing results. Could be an interesting way to increase/decrease difficulty even

i will look into how easy it would be to zip up some subset of the results without leaking anything...probably doable

Post reply on HN