Can we use those metrics in a review pass for every PR? So review agent will have to pinpoint new complexity and propose refactoring or justify increase?
Benchmarking Opus 5 on SlopCodeBench
71–80 of 131 posts
Re: Benchmarking Opus 5 on SlopCodeBench
#72I hope the big labs will start using this benchmark in their RL pipelines. Reducing complexity in generated code should be the number 1 priority, in my opinion. The holy grail for me is models implementing features while reducing LoC (i.e., choosing the right abstractions). What is also nice about this benchmark is that it can be used to iterate on prompts/skills for reducing code complexity.
> I hope the big labs will start using this benchmark in their RL pipelines.
Labs do not train on benchmark data (allegedly). They can train on similar problems, but benchmarks have specific strings in them that labs are supposed to be aggressive in filtering out of their training corpora.
Re: Benchmarking Opus 5 on SlopCodeBench
#73This benchmark makes me worry a bit that people will just ask their model to reimplement everything from scratch once their requirements become more clear.
Re: Benchmarking Opus 5 on SlopCodeBench
#74I hope the big labs will start using this benchmark in their RL pipelines. Reducing complexity in generated code should be the number 1 priority, in my opinion. The holy grail for me is models implementing features while reducing LoC (i.e., choosing the right abstractions). What is also nice about this benchmark is that it can be used to iterate on prompts/skills for reducing code complexity.
It's just as easy to go too far in the direction of shoving too much logic on a single line in the name of reducing lines of code.
Re: Benchmarking Opus 5 on SlopCodeBench
#75This matches my experience of Opus 5 being a nice improvement over Opus 4.8, but not being revolutionary like Fable felt. I’ve now replaced my use of Opus 4.8 xhigh with Opus 5 medium, and I’m using less tokens and it’s quicker. I can understand people being annoyed by its writing style but for getting work done that really doesn’t bother me. I’ve been really enjoying using it.
Can you elaborate on what felt revolutionary to you about Fable?
Re: Benchmarking Opus 5 on SlopCodeBench
#76Earlier quoted context omitted.
Noticed that too. I wonder if these things just degrade over time, perhaps with the way it writes memories about my project as it goes
I feel like a conspiracy theorist but it feels like every new model release from both Anthropic and OpenAI has 1-2 weeks of fantastic performance then a gradual (and sometimes not so gradual) decline in intelligence. It's like they're quantising the model in the background to optimise for available compute/RAM. Which, if I put my MBA hat on, would make perfect sense. Why not halve the required RAM for "90%" of the pe…
And yet: https://marginlab.ai/trackers/claude-code-historical-perform...
There's clearly random variation, but it also shows each model release is just genuinely better. ( With the exception of a heavily degraded week of opus 4.7, which was acknowledged as a problem at the time. )
There's a psychology of getting used to models after being wowed by the new performance. It sets in as your new baseline expectations, and then when it doesn't deliver, it's felt more acutely. When it does deliver, it's just meeting expectations.
Then a new better model comes along and it's a step up again, another wow moment for a week or two until expectations adjust to meet the new baseline.
Re: Benchmarking Opus 5 on SlopCodeBench
#77I wonder to what extent we could steer performance on this benchmark, by providing an adversary model that penalises code duplication and overall lines of code?
Re: Benchmarking Opus 5 on SlopCodeBench
#78Earlier quoted context omitted.
Noticed that too. I wonder if these things just degrade over time, perhaps with the way it writes memories about my project as it goes
I feel like a conspiracy theorist but it feels like every new model release from both Anthropic and OpenAI has 1-2 weeks of fantastic performance then a gradual (and sometimes not so gradual) decline in intelligence. It's like they're quantising the model in the background to optimise for available compute/RAM. Which, if I put my MBA hat on, would make perfect sense. Why not halve the required RAM for "90%" of the pe…
You should. Feel like a conspiracy theorist when saying things like this.
Users are not reliable or consistent model evaluators. Users adapt to models - the moment they get a model that performs better, their expectations rise, their tasks get harder and their prompts get shorter.
"They made the model worse" is PEBKAC in 9 cases out of 10.
Re: Benchmarking Opus 5 on SlopCodeBench
#79Earlier quoted context omitted.
Noticed that too. I wonder if these things just degrade over time, perhaps with the way it writes memories about my project as it goes
I feel like a conspiracy theorist but it feels like every new model release from both Anthropic and OpenAI has 1-2 weeks of fantastic performance then a gradual (and sometimes not so gradual) decline in intelligence. It's like they're quantising the model in the background to optimise for available compute/RAM. Which, if I put my MBA hat on, would make perfect sense. Why not halve the required RAM for "90%" of the pe…
This is also my experience. I don't know if it's because of the quantization theory, or if it's just me getting used to a certain level of coding performance and gradually less tolerant of the mistakes it makes more over time.
Re: Benchmarking Opus 5 on SlopCodeBench
#80So far my 'solution' to this has been periodically run a separate round of whole-codebase code review (preferably Fable) and then rounds of refactoring off the results of that