Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Benchmarking Opus 5 on SlopCodeBench

github.com

Post reply on HN
Print view View original

91–100 of 131 posts

Re: Benchmarking Opus 5 on SlopCodeBench

#91

Post by bodeadly »

adamtaylor_13 wrote: post #88

I may be crucified for asking this but: is there any proof that slop matters beyond our sensibilities as developers? If the code is ugly, but defects are low—does it matter? If the code is hard to read, but clients are happy—should I care? Genuinely asking. Weird times we live in.

Like anything in coding, it depends. LLMs are good at following conventions. If what you're doing has been done a 1000 times before and there is a good bias, it will produce good results. This is why LLMs are so good at the leafy parts and even the branches the leaves are connected to. But there's always something unconventional about a codebase. If there weren't, it wouldn't be worth writing.

Re: Benchmarking Opus 5 on SlopCodeBench

#92

Post by huflungdung »

try-working wrote: post #84
dmrivers wrote: post #82

I always have Claude recite a pledge before starting coding to fix redundant code it notices over time. It does seem to find redundancies, but only when I point out bugs, that's when it goes into fixing mode and actually applies my Don't Repeat Yourself preference from the CLAUDE.md. The original paper cited by this post does try to see if improved prompting will make a big difference in the end using a `plan_first`…

This is just superstition.

[dead]

Re: Benchmarking Opus 5 on SlopCodeBench

#93

Post by thatfunkymunki »

anentropic wrote: post #80

So far my 'solution' to this has been periodically run a separate round of whole-codebase code review (preferably Fable) and then rounds of refactoring off the results of that

yes, this has been my preferred approach as well. Too much risk of very deep local maxima otherwise.

Re: Benchmarking Opus 5 on SlopCodeBench

#94

Post by Majromax »

adamtaylor_13 wrote: post #88

I may be crucified for asking this but: is there any proof that slop matters beyond our sensibilities as developers? If the code is ugly, but defects are low—does it matter? If the code is hard to read, but clients are happy—should I care? Genuinely asking. Weird times we live in.

> I may be crucified for asking this but: is there any proof that slop matters beyond our sensibilities as developers?

That's precisely what this benchmark tries to quantify. Since the benchmark incrementally expands the scope of each problem, 'sloppy' code is code that is hard to later modify.

I know this firsthand: the dumbest coder I've ever worked with was 'myself six months ago'. That jackass never keeps the documentation up to date and hard-codes things that ought to be exposed as configuration.

The SlopCodeBench is an important but early-stage probe in this direction.

Re: Benchmarking Opus 5 on SlopCodeBench

#95

Post by dhorthy »

jdm2212 wrote: post #89
adamtaylor_13 wrote: post #88

I may be crucified for asking this but: is there any proof that slop matters beyond our sensibilities as developers? If the code is ugly, but defects are low—does it matter? If the code is hard to read, but clients are happy—should I care? Genuinely asking. Weird times we live in.

It's the eternal question with any kind of tech debt -- is it worth a little more velocity now in return for medium-to-long term slowdown? And there's no general right answer.

agree, i think the implication is that low quality code is harder to change in the future

Re: Benchmarking Opus 5 on SlopCodeBench

#97

Post by Vgoose »

NitpickLawyer wrote: post #52
Vgoose wrote: post #17

Nice! I actually ran across this paper+benchmark recently, too. It's the first I've found that start to aim at some of the non-functional and longitudinal requirements that I think have always been an important part of writing production code. It's especially relevant now that models are good enough to solve ~most point-in-time problems. Some relevant but disconnected thoughts: - deterministic scores are so nice - wh…

> another signal I've been thinking about and I'm seeing increasingly get brought up is the state space of a system State space of a system AND the way to make it accessible / visible to a model. Many times a model can work magic if it can "see" the state of a system in a way that suits it. That's why sometimes having a cli added to the environment seems like such a big unlock. Because that cli usually takes a comple…

Ive have a definition of state space that is calculated by just the types of the system. It’s a rough approximation of the true state space but it’s convenient (and more reflective of what we actually mean by system state IMO) and types are inherently accessible to the model. I recently put down some thoughts around working with types to improve communication w/ AI that kind of sets this framing up: https://www.alecvo.org/blog/types-with-ai/. The actual definition is still WIP.

Re: Benchmarking Opus 5 on SlopCodeBench

#98

Post by paffdragon »

adamtaylor_13 wrote: post #88

I may be crucified for asking this but: is there any proof that slop matters beyond our sensibilities as developers? If the code is ugly, but defects are low—does it matter? If the code is hard to read, but clients are happy—should I care? Genuinely asking. Weird times we live in.

> is there any proof that slop matters beyond our sensibilities as developers?

I've seen a few papers recently on the topic in terms of LLM, here is two I found by quick googling

* "Does Code Cleanliness Affect Coding Agents? A Controlled Minimal-Pair Study" https://arxiv.org/abs/2605.20049

"Our findings suggest that traditional maintainability principles remain highly relevant in the era of AI-driven development, shaping the computational cost and navigational efficiency of coding agents."

* "Code for Machines, Not Just Humans: Quantifying AI-Friendliness with Code Health Metrics" https://arxiv.org/abs/2601.02200

"Our findings confirm that human-friendly code is also more compatible with AI tooling."

"Investing in maintainability not only helps humans; it also prepares for large-scale AI adoption."

> If the code is ugly, but defects are low—does it matter? > If the code is hard to read, but clients are happy—should I care?

For you who wrote it and are the sole developer, maybe not. But if you want to have other contributors or hand it off, then it can be a problem. New devs joining the project may not want to work with it and will push for a rewrite that will cost money or need to spend extra time on working with code that is hard to work with for them which will cost money. And from the papers above it seems this also affects LLMs as well as they seem to work more efficiently with "cleaner" code.

Re: Benchmarking Opus 5 on SlopCodeBench

#99

Post by jerf »

WilcoKruijer wrote: post #67

I hope the big labs will start using this benchmark in their RL pipelines. Reducing complexity in generated code should be the number 1 priority, in my opinion. The holy grail for me is models implementing features while reducing LoC (i.e., choosing the right abstractions). What is also nice about this benchmark is that it can be used to iterate on prompts/skills for reducing code complexity.

I would at least consider the possibility that they already are, and this is how it is going. It could be a fundamental architecture problem for LLMs. It's not like "slop" is a new problem and I'm sure they'd love to announce a new model that generates much less "slop". The fact they've never so much as mentioned it suggests to me that it's not something they've been able to fix.
Post reply on HN