Measuring the sloppiness of code • Hacker News Board

Re: Measuring the sloppiness of code

#141

Post by econ » Fri, Sep 11, 2026, 3:42 PM UTC

whatever1 wrote: post #82

mr_roboto wrote: post #74

Earlier quoted context omitted.

You come across as someone who has never worked on a real software project. Humans create tons of bugs on a regular basis. AI is already better than most programmers.

Better at writing one piece of code, maybe. Better at writing code within a huge system, definitely not. Maybe in the future, but as of Astra, Fable 5.1, the answer is still no.

I'm definitely not experienced enough to know but I read people are having it write somewhat elaborate documentation beforehand. Have it figure out which parts of the code may be touched, what other things will be affected and which uncertainties it has. Basically a full report before giving permission for any code to be written/modified. Again, I haven't seen it, don't know how common this is nor how effective. Though it sounded interesting.

Re: Measuring the sloppiness of code

#142

Post by farhadhf » Fri, Sep 11, 2026, 3:42 PM UTC

Coding is not solved. It's only solved when coding becomes something you do because you want to, just because you like doing it, the same way I bake bread at home because I like doing it, not because I have to. Right now we still have to be hands-on - to a lesser extent, yes - but we still have to review and hand holding AI agents to get things done.

Re: Measuring the sloppiness of code

#143

Post by danielvaughn » Fri, Sep 11, 2026, 3:48 PM UTC

margalabargala wrote: post #129

heaney-555 wrote: post #92

Earlier quoted context omitted.

Plenty of serious production projects are doing exactly that. Are you using GPT-6 Astra, or something older?

You and the person you are replying to are talking about different things. Agents cannot be given a high level goal and then left unsupervised, for hours, without making some dumb decisions.

Right. The only way I'd feel comfortable doing that is if I spent an inordinate amount of time writing very detailed specs, so in terms of labor offset I'm not sure the juice would even be worth the squeeze there. In other words, a sufficiently detailed spec is nearly indistinguishable from code.

What people seem to be wanting is for an agent to infer vast complex data from terse simple data, which I think is probably impossible on a philosophical level. There's real information loss in language, and compute can only make guesses at the end of the day. I really don't see how we bridge that gap.

Re: Measuring the sloppiness of code

#144

Post by flyinglizard » Fri, Sep 11, 2026, 3:59 PM UTC

einrealist wrote: post #106

danbruc wrote: post #19

Coding is not solved, correctness is not a feature, it is the bare minimum. If your code does not do what it is supposed to do, you could as well have no code at all. Efficiency, security, maintainability, reliability, readability, understandability, extensibility, maintainability, observability, portability, ... this is what high quality coding is about, not that it works, that is a given. And in my experience curre…

And there is another problem: LLMs generating too much code, code that is doing more than was asked. And that cannot be fixed by tests. Usually, we create tests for wanted behavior and expected exceptions. But we don't create tests for undesired behavior.

Code cost is almost down to zero. If you move the point of “just leave it to the machine” from the compiler (where humans used to do the coding) to the high level logic (now with LLMs) then in most cases more code does not really matter. Like, why build and maintain an abstraction where the LLM could implement this many times over each time with different subtleties? Why use a library with its own constraints when you could have exactly what you want? Why use cross platform frameworks when you can just one shot the thing to N different platforms? It’s not even slower. You can have code that’s larger yet more performant (stripping away abstractions can do that).

From time to time I try to do a pass of coalescing flows and cases and removing dead code to reduce the context and prevent the LLM from tripping over itself. But if it’s exclusively LLM maintained code I don’t care too much if there’s more of it.

Re: Measuring the sloppiness of code

#145

Post by mikkelam » Fri, Sep 11, 2026, 4:00 PM UTC

This is once again RLHF loops.

the AI labs are and have been 100% focused on correctness because it is easy to setup and validate.

Adding one more function that almost does the same thing as another will not break anything.

I think this is just a matter of time. At some point there'll be less value to squeeze out of correctness and then the AI labs will start focusing on maintainability. It's probably a lot harder to set up environment to Train for this behavior though.

Re: Measuring the sloppiness of code

#146

Post by gpugreg » Fri, Sep 11, 2026, 4:02 PM UTC

cjalmeida wrote: post #31

>In my research and tests simply taking the change in the number of LOCs has been a surprisingly effective metric for sloppiness, with the ironic caveat that if we started optimizing for it, it would cease to be a meaningful measure. This matches my experience. Before working on an issue, I ask the LLM to estimate net LOCs at the final PR based on the scope. It works well, and review steps do flag inconsistencies. Bu…

I've had some success with tokens as a measure of complexity instead of number of lines, but should be combined with additional rules, e.g. disallowing lambdas, exec, eval, compile, __import__ and complex list comprehensions for Python. Fortunately, Python's "ast" module makes this quite easy.

Re: Measuring the sloppiness of code

#147

Post by kloud » Fri, Sep 11, 2026, 4:04 PM UTC

The problem is there is no good metric to describe code quality, it cannot be RLd and that's likely why latest models have such problems with slop. It is great to raise the conversation so that labs focus on this more.

Picking specific metrics will probably not work, it would be a mix of Goodhart's law with Bitter lesson. Maybe picking and labeling quality repos, having whole suite of metrics as input features and training some traditional AI classifiers to steer the LLM training.

Re: Measuring the sloppiness of code

#148

Post by bigstrat2003 » Fri, Sep 11, 2026, 4:04 PM UTC

danbruc wrote: post #19

Coding is not solved, correctness is not a feature, it is the bare minimum. If your code does not do what it is supposed to do, you could as well have no code at all. Efficiency, security, maintainability, reliability, readability, understandability, extensibility, maintainability, observability, portability, ... this is what high quality coding is about, not that it works, that is a given. And in my experience curre…

Yeah, the author leads with a blatantly false claim that models are almost perfect at generating code. They are not. They write pretty bad code, worse than any capable human would.

Re: Measuring the sloppiness of code

#149

Post by anon-3988 » Fri, Sep 11, 2026, 4:05 PM UTC

danbruc wrote: post #19

Coding is not solved, correctness is not a feature, it is the bare minimum. If your code does not do what it is supposed to do, you could as well have no code at all. Efficiency, security, maintainability, reliability, readability, understandability, extensibility, maintainability, observability, portability, ... this is what high quality coding is about, not that it works, that is a given. And in my experience curre…

Look, we are comparing against the average developer here And yes, they are obsolete

Re: Measuring the sloppiness of code

#150

Post by batshit_beaver » Fri, Sep 11, 2026, 4:05 PM UTC

mr_roboto wrote: post #99

danbruc wrote: post #91

Earlier quoted context omitted.

I have worked my entire life as a professional software developer and I agree, even among senior developers I would guess [way] less then 10 % consistently produce high quality code. But if I have to decided whether I want to use an AI to help me write code, I does not matter if it can write better code then an unexperienced junior, it has to write better code than I would on my own.

I've found I can produce 10x more code than I could otherwise, of lower quality than I would otherwise, but the speedup is worth it. Extensive testing is what makes it work, with every bug becoming a red first test with a fix. High level compartmentalization keeps everything on track, you don't let it do the big picture architecture, but you let it do each component as decided on and work through the bugs later. I've…

> I've found I can produce 10x more code than I could otherwise, of lower quality than I would otherwise, but the speedup is worth it.

The speedup of slop production being “worth it” is what we, as a society, are having trouble evaluating at this point in time. In all likelihood it’s worth it only in the short term.