Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Measuring the sloppiness of code

earendil.com

Post reply on HN
Print view View original

151–160 of 247 posts

Re: Measuring the sloppiness of code

#151

Post by bigstrat2003 »

mr_roboto wrote: post #99
danbruc wrote: post #91

Earlier quoted context omitted.

I have worked my entire life as a professional software developer and I agree, even among senior developers I would guess [way] less then 10 % consistently produce high quality code. But if I have to decided whether I want to use an AI to help me write code, I does not matter if it can write better code then an unexperienced junior, it has to write better code than I would on my own.

I've found I can produce 10x more code than I could otherwise, of lower quality than I would otherwise, but the speedup is worth it. Extensive testing is what makes it work, with every bug becoming a red first test with a fix. High level compartmentalization keeps everything on track, you don't let it do the big picture architecture, but you let it do each component as decided on and work through the bugs later. I've…

> I've found I can produce 10x more code than I could otherwise, of lower quality than I would otherwise, but the speedup is worth it.

That is never worth it. You're ruining the software you work on when you do this.

Re: Measuring the sloppiness of code

#152

Post by guywithahat »

scronkfinkle wrote: post #12

There is some sense of rose-tinted glasses of pre-LLM coding. A lot of human written code, particularly at the enterprise level, was of low quality well before AI automated it.

I would argue the average code quality of LLM's today is much higher than pre-AI code quality. It's better documented, more readable, and has fewer bugs. There was a brief period where frontier models were still worse than the average developer, but that period among frontier models is well past us.

Re: Measuring the sloppiness of code

#153

Post by guywithahat »

Varelion wrote: post #86
Dlemlo wrote: post #78

Earlier quoted context omitted.

So they still might do but in parallel the software engineering market in china and india is collapsing.

Source?

I think he's referring to contracting companies. Cheap software used to be outsourced to India, but it's now getting outsourced to AI. Companies that produce novel things are still hiring, often to manage agents and review output.

Re: Measuring the sloppiness of code

#154

Post by brap »

mikkelam wrote: post #145

This is once again RLHF loops. the AI labs are and have been 100% focused on correctness because it is easy to setup and validate. Adding one more function that almost does the same thing as another will not break anything. I think this is just a matter of time. At some point there'll be less value to squeeze out of correctness and then the AI labs will start focusing on maintainability. It's probably a lot harder to…

There’s one thing I constantly see agents tripping over, I’m not sure what the right word for it would be, but it basically boils down to “making changes in the right places”. They seem to have very poor grasp of where things are supposed to be and they have a tendency to work against the existing architecture. Even in a world where agents are the only ones touching the code you can see how this ends poorly. Unlike correctness I’m not sure there’s an easy way to verify.

I tried writing a few skills to encourage agents to spend time thinking about this but it doesn’t seem to generalize very well.

Re: Measuring the sloppiness of code

#155

Post by bigstrat2003 »

echelon wrote: post #29
danbruc wrote: post #19

Coding is not solved, correctness is not a feature, it is the bare minimum. If your code does not do what it is supposed to do, you could as well have no code at all. Efficiency, security, maintainability, reliability, readability, understandability, extensibility, maintainability, observability, portability, ... this is what high quality coding is about, not that it works, that is a given. And in my experience curre…

And all of these things will be solved one by one. It's astounding to me that people can see coding get solved and not think every single one of these tasks won't be solved too. Why do you not think these things aren't going to be completely automated? What makes these tasks special? Fable and Astra can one-shot video games with compelling novel game loops. They can do systems programming, distributed systems, roboti…

> It's astounding to me that people can see coding get solved...

We haven't seen that. Maybe when we do, we will start to believe that other things will get solved.

Re: Measuring the sloppiness of code

#156

Post by Rover222 »

I'm not shocked the top 2 comments are "coding is not solved"

Pure cope, at least in most domains. The machines can do it better than the humans.

I think most arguments against that are clinging onto old things that don't really matter any more, like coding style.

If most of us are not coding by hand at all anymore, just directing and iterating with the agents, then coding is solved by a reasonable definition.

Re: Measuring the sloppiness of code

#157

Post by LevGoldstein »

danielvaughn wrote: post #68
danbruc wrote: post #19

Coding is not solved, correctness is not a feature, it is the bare minimum. If your code does not do what it is supposed to do, you could as well have no code at all. Efficiency, security, maintainability, reliability, readability, understandability, extensibility, maintainability, observability, portability, ... this is what high quality coding is about, not that it works, that is a given. And in my experience curre…

Yes. I use these models day in and day out, on all sorts of tasks. I cannot believe I hear people say that coding is solved.

Consider that the people who have never have never had to be on-call for a product that results in losses for the company in the event of an outage, and/or have never had to attend an RCA meeting have just as much of a voice in this discussion. Some segments of the software industry don't even have to worry about those sorts of things, and it's easy to get myopic after a while...I doubt that a long time dev at hobbyist-centric wearable manufacturers or audio plugin shops has the same sorts of concerns as someone who has specialized in ad delivery for social media sites.

Re: Measuring the sloppiness of code

#158

Post by jimbokun »

LevGoldstein wrote: post #157
danielvaughn wrote: post #68

Earlier quoted context omitted.

Yes. I use these models day in and day out, on all sorts of tasks. I cannot believe I hear people say that coding is solved.

Consider that the people who have never have never had to be on-call for a product that results in losses for the company in the event of an outage, and/or have never had to attend an RCA meeting have just as much of a voice in this discussion. Some segments of the software industry don't even have to worry about those sorts of things, and it's easy to get myopic after a while...I doubt that a long time dev at hobbyi…

Allowing only people who have served on call or made concrete contributions to multiple RCAs to have input on the software development process is a wonderful idea!

Re: Measuring the sloppiness of code

#159

Post by mwigdahl »

bla3 wrote: post #104
Supermancho wrote: post #66

Earlier quoted context omitted.

> Efficiency, security, maintainability, reliability, readability, understandability, extensibility, maintainability, observability, portability My experience is a little different. For higher abstraction languages the output is largely acceptable in my work. I always consider that LLMs don't know what I don't tell them and they have limited context to work from. Coding issues I often identify: * Efficiency. Marginal…

This sounds roughly right to me, except for "maintainability". In my experience, agents really don't like deleting code unless you explicitly ask for it. If you're not careful, you end up with new better implementations of things but with the old implementation still around in perpetuity. Humans do this too of course.

They're being trained by a lot of repetitions of one-shot bugfixes and feature implementations. If they don't have to maintain code over the long term they have no incentive to not take the easy win with lower immediate risk and higher longer-term maintenance burden.

Re: Measuring the sloppiness of code

#160

Post by mwigdahl »

technoplato wrote: post #93

Plopping in my email to the author below in case anyone else is interested in this kind of thing: > There are some promising other directions I want to explore, such as coupledness of functions, code churn, cohesion and so on. If you are working on evals and would like to talk, I would be happy to do that: sebastian@earendil.com Hey Sebastian, I just read your article and it thoroughly resonated with me. I've been wo…

That's a very interesting idea for a benchmark!
Post reply on HN