Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Benchmarking Opus 5 on SlopCodeBench

github.com

Post reply on HN
Print view View original

51–60 of 131 posts

Re: Benchmarking Opus 5 on SlopCodeBench

#52

Post by NitpickLawyer »

Vgoose wrote: post #17

Nice! I actually ran across this paper+benchmark recently, too. It's the first I've found that start to aim at some of the non-functional and longitudinal requirements that I think have always been an important part of writing production code. It's especially relevant now that models are good enough to solve ~most point-in-time problems. Some relevant but disconnected thoughts: - deterministic scores are so nice - wh…

> another signal I've been thinking about and I'm seeing increasingly get brought up is the state space of a system

State space of a system AND the way to make it accessible / visible to a model. Many times a model can work magic if it can "see" the state of a system in a way that suits it. That's why sometimes having a cli added to the environment seems like such a big unlock. Because that cli usually takes a complex state and allows visibility into it, and possible manipulation in a structured way.

Re: Benchmarking Opus 5 on SlopCodeBench

#53

Post by jbird99 »

rubicon33 wrote: post #22
sothatsit wrote: post #18

This matches my experience of Opus 5 being a nice improvement over Opus 4.8, but not being revolutionary like Fable felt. I’ve now replaced my use of Opus 4.8 xhigh with Opus 5 medium, and I’m using less tokens and it’s quicker. I can understand people being annoyed by its writing style but for getting work done that really doesn’t bother me. I’ve been really enjoying using it.

I think they neutered Fable. When it first came out it was indeed revolutionary. But what we have today, is not what we had before the ban.

At this point I don't even bother with it. Constantly falls back to Opus anyway, so I may as well save myself some time.

Re: Benchmarking Opus 5 on SlopCodeBench

#55

Post by piazz »

Great writeup. The excessive function thing has always driven me crazy; I guard against this explicitly in Claude.md.

I have found that models are generally poor at managing refactors / complexity while also implementing new features. But I’ve had some success with a semi-lights-off approach where you decompose it and prompt the model adversarially in a second pass to look for new rough edges and areas of complexity or refactors that might simplify the codebase.

So I’d be very curious to see this benchmark but with something like a periodic “refactor turn” interleaved in.

Also eager to see Fable benchmarked; anecdotally that was the only model whose code I felt I could actually trust to not review closely.

Re: Benchmarking Opus 5 on SlopCodeBench

#57

Post by nerdsniper »

conception wrote: post #37
Espressosaurus wrote: post #29

Earlier quoted context omitted.

I’ve observed the degradation, but I suspect what’s happening is they’re tuning it for lower inference costs. Maybe turning down the amount of thinking, maybe quantizing, maybe something else. It seems like there’s a week by week and sometimes day by day change in performance when on a subscription plan using their harnesses.

https://marginlab.ai/trackers/claude-code/ their tracker generally shows that isn’t the case. The only times I’ve seen it drop is something broken and just before fable launched.

I mean they could just be routing known benchmark questions (which all of SWEBench are) to a full-performance variant.
Post reply on HN