Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Benchmarking Opus 5 on SlopCodeBench

github.com

Post reply on HN
Print view View original

21–30 of 131 posts

Re: Benchmarking Opus 5 on SlopCodeBench

#21

Post by fastball »

sothatsit wrote: post #18

This matches my experience of Opus 5 being a nice improvement over Opus 4.8, but not being revolutionary like Fable felt. I’ve now replaced my use of Opus 4.8 xhigh with Opus 5 medium, and I’m using less tokens and it’s quicker. I can understand people being annoyed by its writing style but for getting work done that really doesn’t bother me. I’ve been really enjoying using it.

Medium vs High? Why? From all the charts I've seen the performance jump is pretty large from med -> high (not as noticeable from high -> xhigh).

Re: Benchmarking Opus 5 on SlopCodeBench

#22

Post by rubicon33 »

sothatsit wrote: post #18

This matches my experience of Opus 5 being a nice improvement over Opus 4.8, but not being revolutionary like Fable felt. I’ve now replaced my use of Opus 4.8 xhigh with Opus 5 medium, and I’m using less tokens and it’s quicker. I can understand people being annoyed by its writing style but for getting work done that really doesn’t bother me. I’ve been really enjoying using it.

I think they neutered Fable. When it first came out it was indeed revolutionary. But what we have today, is not what we had before the ban.

Re: Benchmarking Opus 5 on SlopCodeBench

#23

Post by ajwin »

To what degree is this a harness/system prompt problem? Models maybe should implement new stuff with as little impact on the existing stuff as possible by default? A simple system prompt for it to always check the code after task completion for proper simplifications, abstractions and cleanups before returning to the user? Instructions to retain "story like" readability of the code.

Re: Benchmarking Opus 5 on SlopCodeBench

#24

Post by [deleted] »

sothatsit wrote: post #18

This matches my experience of Opus 5 being a nice improvement over Opus 4.8, but not being revolutionary like Fable felt. I’ve now replaced my use of Opus 4.8 xhigh with Opus 5 medium, and I’m using less tokens and it’s quicker. I can understand people being annoyed by its writing style but for getting work done that really doesn’t bother me. I’ve been really enjoying using it.

[deleted]

Re: Benchmarking Opus 5 on SlopCodeBench

#25

Post by dhorthy »

ajwin wrote: post #23

To what degree is this a harness/system prompt problem? Models maybe should implement new stuff with as little impact on the existing stuff as possible by default? A simple system prompt for it to always check the code after task completion for proper simplifications, abstractions and cleanups before returning to the user? Instructions to retain "story like" readability of the code.

I agree this is an option, and the next thing on my radar is to try with a more realistic "factory-shaped" harness where you have feedback from linters and other models after each coding episode that refines the architecture.

For readability specifically, I've found it hard to get the models to do this with prompting. If you've talked to opus/fable for a long time on prose writing you probably felt this too

Re: Benchmarking Opus 5 on SlopCodeBench

#26

Post by igregoryca »

ajwin wrote: post #23

To what degree is this a harness/system prompt problem? Models maybe should implement new stuff with as little impact on the existing stuff as possible by default? A simple system prompt for it to always check the code after task completion for proper simplifications, abstractions and cleanups before returning to the user? Instructions to retain "story like" readability of the code.

At least for Claude Code, putting "run /simplify at the end" in an "implement the plan" skill helps a little. It still often leaves new code in bizarre places, and/or with bad/alien-sounding names and comments.

Re: Benchmarking Opus 5 on SlopCodeBench

#27

Post by sothatsit »

fastball wrote: post #21
sothatsit wrote: post #18

This matches my experience of Opus 5 being a nice improvement over Opus 4.8, but not being revolutionary like Fable felt. I’ve now replaced my use of Opus 4.8 xhigh with Opus 5 medium, and I’m using less tokens and it’s quicker. I can understand people being annoyed by its writing style but for getting work done that really doesn’t bother me. I’ve been really enjoying using it.

Medium vs High? Why? From all the charts I've seen the performance jump is pretty large from med -> high (not as noticeable from high -> xhigh).

If I need something smarter I use Fable. Medium works well and is quick. Opus 5 medium feels much better to me than Opus 4.8 medium.

Re: Benchmarking Opus 5 on SlopCodeBench

#28

Post by swader999 »

rubicon33 wrote: post #22
sothatsit wrote: post #18

This matches my experience of Opus 5 being a nice improvement over Opus 4.8, but not being revolutionary like Fable felt. I’ve now replaced my use of Opus 4.8 xhigh with Opus 5 medium, and I’m using less tokens and it’s quicker. I can understand people being annoyed by its writing style but for getting work done that really doesn’t bother me. I’ve been really enjoying using it.

I think they neutered Fable. When it first came out it was indeed revolutionary. But what we have today, is not what we had before the ban.

Noticed that too. I wonder if these things just degrade over time, perhaps with the way it writes memories about my project as it goes

Re: Benchmarking Opus 5 on SlopCodeBench

#29

Post by Espressosaurus »

swader999 wrote: post #28
rubicon33 wrote: post #22

Earlier quoted context omitted.

I think they neutered Fable. When it first came out it was indeed revolutionary. But what we have today, is not what we had before the ban.

Noticed that too. I wonder if these things just degrade over time, perhaps with the way it writes memories about my project as it goes

I’ve observed the degradation, but I suspect what’s happening is they’re tuning it for lower inference costs. Maybe turning down the amount of thinking, maybe quantizing, maybe something else.

It seems like there’s a week by week and sometimes day by day change in performance when on a subscription plan using their harnesses.

Re: Benchmarking Opus 5 on SlopCodeBench

#30

Post by dhorthy »

sothatsit wrote: post #27
fastball wrote: post #21

Earlier quoted context omitted.

Medium vs High? Why? From all the charts I've seen the performance jump is pretty large from med -> high (not as noticeable from high -> xhigh).

If I need something smarter I use Fable. Medium works well and is quick. Opus 5 medium feels much better to me than Opus 4.8 medium.

yeah someone will have to re-run this bench on various effort levels. unfortunately it is not cheap
Post reply on HN