Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Measuring the sloppiness of code

earendil.com

Post reply on HN
Print view View original

21–30 of 246 posts

Re: Measuring the sloppiness of code

#21

Post by marginalia_nu »

victorbjorklund wrote: post #15
scronkfinkle wrote: post #12

There is some sense of rose-tinted glasses of pre-LLM coding. A lot of human written code, particularly at the enterprise level, was of low quality well before AI automated it.

Indeed. And not fair comparisons ”look at the quality of this small one-shot Claude hobby project. The quality is less than this major open source project written by some of the best developers in the world”

To be fair the pitch has frequently been that Devin/Claude/Astra/whatever is some sort of superhuman bottled John Carmack that will single-handedly replace entire teams of developers.

Re: Measuring the sloppiness of code

#22

Post by softwaredoug »

scronkfinkle wrote: post #12

There is some sense of rose-tinted glasses of pre-LLM coding. A lot of human written code, particularly at the enterprise level, was of low quality well before AI automated it.

I don't doubt that. But humans still need to be responsible for understanding what they're shipping. And IMO you get your best understanding by actually writing some code. Even if you don't actually ship what you wrote.

Re: Measuring the sloppiness of code

#24

Post by alecbz »

peder wrote: post #7
fosterfriends wrote: post #3

I love this train of thought. Code quality is critical, but I don’t think we’re correctly evaling it at the moment. If we could get solid benchmarks measuring the quality of generated code, we might see the models climb those benches fast. I believe that the era of “ai writes tons of slop code” will be a stepping stone in the longer story, and is simply a current gap in the reward functions. Per the author - if we ca…

Sooooo much of what is considered "code quality" today is irrelevant when robots are writing the code. We've been largely optimizing for things like composability/unit testability in the past 15 or so years, and that's primarily a human concern that's unrelated to the final output. Totally agreed that we're not looking at the correct metric right now. Increasingly, code quality will be determined by outcomes.

> that's primarily a human concern that's unrelated to the final output.

It's an LLM concern too. LLMs seem to do better with well-organized codebases, just like the humans they were trained on.

Re: Measuring the sloppiness of code

#26

Post by zsoltkacsandi »

danbruc wrote: post #19

Coding is not solved, correctness is not a feature, it is the bare minimum. If your code does not do what it is supposed to do, you could as well have no code at all. Efficiency, security, maintainability, reliability, readability, understandability, extensibility, maintainability, observability, portability, ... this is what high quality coding is about, not that it works, that is a given. And in my experience curre…

That is very well put and summarizes what distinguishes real software development from vibe coding.

Re: Measuring the sloppiness of code

#27

Post by stiiv »

peder wrote: post #7
fosterfriends wrote: post #3

I love this train of thought. Code quality is critical, but I don’t think we’re correctly evaling it at the moment. If we could get solid benchmarks measuring the quality of generated code, we might see the models climb those benches fast. I believe that the era of “ai writes tons of slop code” will be a stepping stone in the longer story, and is simply a current gap in the reward functions. Per the author - if we ca…

Sooooo much of what is considered "code quality" today is irrelevant when robots are writing the code. We've been largely optimizing for things like composability/unit testability in the past 15 or so years, and that's primarily a human concern that's unrelated to the final output. Totally agreed that we're not looking at the correct metric right now. Increasingly, code quality will be determined by outcomes.

I think Martin F's team tied outcomes to factoring https://martinfowler.com/articles/exploring-gen-ai/refactori... which is crucial to code quality. Even grug brain developer agree.

On the other hand, there are claims that the best languages for robots tend to be terse (allegedly); I tend to regard a rich domain model (static types, not terse) as a major facet of high-quality code.

Re: Measuring the sloppiness of code

#28

Post by Kinrany »

Number of iterations solved correctly, on a very large set of iterations, seems like a very good metric. Better than anything else because it measures what we actually care about, not some proxy.

The only caveat is that it's the same model doing an iteration and then using that iteration as a starting point for the next step. So the model is allowed to write absolutely insane solutions, as long as it can read them back, even if no one else can.

One thing that could be done is to use a separately developed baseline coding model B to evaluate the outcome of each iteration. For model under test X to pass an iteration, not only should it be able to solve starting from the previous solution, but so should B, starting from X's previous solution.

Re: Measuring the sloppiness of code

#29

Post by echelon »

danbruc wrote: post #19

Coding is not solved, correctness is not a feature, it is the bare minimum. If your code does not do what it is supposed to do, you could as well have no code at all. Efficiency, security, maintainability, reliability, readability, understandability, extensibility, maintainability, observability, portability, ... this is what high quality coding is about, not that it works, that is a given. And in my experience curre…

And all of these things will be solved one by one.

It's astounding to me that people can see coding get solved and not think every single one of these tasks won't be solved too.

Why do you not think these things aren't going to be completely automated? What makes these tasks special?

Fable and Astra can one-shot video games with compelling novel game loops. They can do systems programming, distributed systems, robotics. I haven't found a weak point.

Seedance 2.5 can make video better than the manual labor of VFX artists, 3D artists, and animators.

Nano Banana and GPT Image can do a better job than graphics designers.

LLMs just solved a Millennium Prize Problem, and there are probably more that will fall in the coming weeks.

Just wait. All of these things will be solved.

There is no "stopping point".

Edit:

Don't anticipate that 2036 will look anything like 2026.

Will Smith spaghetti doesn't stay that way forever. Trillions of dollars will be spent on solving these problems. They will be solved.

Post reply on HN