>In my research and tests simply taking the change in the number of LOCs has been a surprisingly effective metric for sloppiness, with the ironic caveat that if we started optimizing for it, it would cease to be a meaningful measure.
This matches my experience. Before working on an issue, I ask the LLM to estimate net LOCs at the final PR based on the scope. It works well, and review steps do flag inconsistencies. But as the OP mentioned, if you turn this into a hard metric vs "design smell", you can see LLMs code-golfing for oneliners.
I love this train of thought. Code quality is critical, but I don’t think we’re correctly evaling it at the moment. If we could get solid benchmarks measuring the quality of generated code, we might see the models climb those benches fast. I believe that the era of “ai writes tons of slop code” will be a stepping stone in the longer story, and is simply a current gap in the reward functions. Per the author - if we ca…
Sooooo much of what is considered "code quality" today is irrelevant when robots are writing the code. We've been largely optimizing for things like composability/unit testability in the past 15 or so years, and that's primarily a human concern that's unrelated to the final output. Totally agreed that we're not looking at the correct metric right now. Increasingly, code quality will be determined by outcomes.
I don’t agree with this. The things people care about with code quality if you really think about it actually match up surprisingly well with the metric which models are trained to emulate in pre training, namely compression and modularity. Those two ideas actually seem to be universal to intelligent systems. Writing verbose highly coupled code is I think provably stupid, though I don’t know that I could formalize it.
There is some sense of rose-tinted glasses of pre-LLM coding. A lot of human written code, particularly at the enterprise level, was of low quality well before AI automated it.
1. People didn’t wear that as a badge of honour though.
2. A lot of it wasn’t. Low quality code/speed serves a purpose for point solutions and scripts etc. That’s not the same thing as writing a core system and if the user doesn’t put any credentials in for an S3 bucket then it falls back to giving information about your own S3 bucket (as I’ve seen just this week).
3. Plenty of companies you can discern the difference between mission critical systems versus “business” systems where if it falls over it’s annoying but not the end of the world.
There is some sense of rose-tinted glasses of pre-LLM coding. A lot of human written code, particularly at the enterprise level, was of low quality well before AI automated it.
I don't doubt that. But humans still need to be responsible for understanding what they're shipping. And IMO you get your best understanding by actually writing some code. Even if you don't actually ship what you wrote.
Let’s not romanticize it too much... A lot of enterprise systems are built by developers copying an old AbstractBeanFactoryFactory from a 2011 stack overflow thread without really understanding it :)
Ya, now that I have some solid AI coding experience under my belt, there does seem to be some gaps between practice and reality. I have a fairly complex codebase which I pretty much hand code everything. When I add a new feature, I spend a lot of time designing and refactoring that feature into the codebase. Either the feature dovetails into the existing design or the feature creates new designs which will then facilitate even better future features. When AI approaches the feature, it just plows the feature in, and with bugs since it has trouble fully understanding the total design. So over time, you have a spaghetti design where you just have a whole bunch of features tied together with no unified design. I guess thats ok if AI is supporting it, but you now have a large cost and bug surface area and an insane human learning curve. So nothing has really changed here, we have been dealing with low quality codebases way before AI came along. I think AI has mastered the one shot single feature, tool, or simple app, but it struggles with the design complexity of a rich multi feature application or system.
There is some sense of rose-tinted glasses of pre-LLM coding. A lot of human written code, particularly at the enterprise level, was of low quality well before AI automated it.
I don't doubt that. But humans still need to be responsible for understanding what they're shipping. And IMO you get your best understanding by actually writing some code. Even if you don't actually ship what you wrote.
> But humans still need to be responsible for understanding what they're shipping
I don't necessarily disagree. That said...
Why?
I've been grappling with this myself. There is an easy/obvious answer, but I wonder how stable/permanent it is. If you feel strongly about this, are you willing to unpack your judgement?
What has this author written before LLMs? Why should we listen to him and his adjudication of "perfect code"?
Cyclomatic complexity is the oldest paper generating grift for college students. There are hundreds of thousands of useless papers about cyclomatic complexity.
Coding is not solved, correctness is not a feature, it is the bare minimum. If your code does not do what it is supposed to do, you could as well have no code at all. Efficiency, security, maintainability, reliability, readability, understandability, extensibility, maintainability, observability, portability, ... this is what high quality coding is about, not that it works, that is a given. And in my experience curre…
Amen. I sometimes wonder if all programmers are now marketing people who know shit about software development and engineering.