Measuring the sloppiness of code
earendil.com
Measuring the sloppiness of code
1–10 of 246 posts
Re: Measuring the sloppiness of code
#2"To come back to the point of why agents can’t (really) deal with the slop themselves, we need to look at the evaluation of SlopCodeBench. In contrast to other coding benchmarks, which give the agent a complete list of instructions at the start and then have a set of hidden tests the program needs to pass, they do the opposite. They create multiple rounds of instruction and test iterations, where in between checkpoints the context of the models is erased. Thereby mimicking much more closely an iterative process, like how coding agents are actually used by humans. The result of that is that bad coding decisions accumulate over time and for the strict solve rate, where all tests have to be passed at all checkpoints, even state of the art models achieve 0% pass rate"
I like how this captures with a metric (lines of code and cyclomatic complexity, some other basically tractable measures) in an automated way, something we all by now intuitively know.
Re: Measuring the sloppiness of code
#3I love this train of thought. Code quality is critical, but I don’t think we’re correctly evaling it at the moment. If we could get solid benchmarks measuring the quality of generated code, we might see the models climb those benches fast.
I believe that the era of “ai writes tons of slop code” will be a stepping stone in the longer story, and is simply a current gap in the reward functions.
Per the author - if we can get strong measurements of what good code is, we can train against it and close the gap fast. Excited to see more thinking in this area
Re: Measuring the sloppiness of code
#4It is not clear to me how the verbosity metrics works. Can someone shades more light on it?
Re: Measuring the sloppiness of code
#5I love this train of thought. Code quality is critical, but I don’t think we’re correctly evaling it at the moment. If we could get solid benchmarks measuring the quality of generated code, we might see the models climb those benches fast. I believe that the era of “ai writes tons of slop code” will be a stepping stone in the longer story, and is simply a current gap in the reward functions. Per the author - if we ca…
If you could easily benchmark the quality of code then models would be trained on these benchmarks/metrics.
Re: Measuring the sloppiness of code
#6Well... Coding was never the most part of work I spent time.
Re: Measuring the sloppiness of code
#7I love this train of thought. Code quality is critical, but I don’t think we’re correctly evaling it at the moment. If we could get solid benchmarks measuring the quality of generated code, we might see the models climb those benches fast. I believe that the era of “ai writes tons of slop code” will be a stepping stone in the longer story, and is simply a current gap in the reward functions. Per the author - if we ca…
Sooooo much of what is considered "code quality" today is irrelevant when robots are writing the code. We've been largely optimizing for things like composability/unit testability in the past 15 or so years, and that's primarily a human concern that's unrelated to the final output.
Totally agreed that we're not looking at the correct metric right now. Increasingly, code quality will be determined by outcomes.
Re: Measuring the sloppiness of code
#8Coding might be "solved" but coding still is the best way to build your own mental model of the solution space.
Which is more important to you: Velocity to a solution? Or velocity to understanding?
Re: Measuring the sloppiness of code
#9I love this train of thought. Code quality is critical, but I don’t think we’re correctly evaling it at the moment. If we could get solid benchmarks measuring the quality of generated code, we might see the models climb those benches fast. I believe that the era of “ai writes tons of slop code” will be a stepping stone in the longer story, and is simply a current gap in the reward functions. Per the author - if we ca…
If you could easily benchmark the quality of code then models would be trained on these benchmarks/metrics.
that is true, but if the metric is what we want optimized, then that's fine.
However it is more likely to be something which can be detached..
Re: Measuring the sloppiness of code
#10Well... Coding was never the most part of work I spent time.
Congrats on your promotion to full time jira management!