Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Coding on Copilot: Data suggests downward pressure on code quality

gitclear.com

Post reply on HN
Print view View original

11–20 of 60 posts

Re: Coding on Copilot: Data suggests downward pressure on code quality

#11

Post by hedora »

dartos wrote: post #7
tigeroil wrote: post #4

Earlier quoted context omitted.

Indeed - if anything in my organisation AI has lead to developers replacing poor quality existing code more quickly because what was quite a time consuming process before can be sped up tremendously by using language models.

Replacing poor code, with poor code that nobody really understands isn’t good either. Imagine hiring a junior to write your code, except they’d just stop existing when they were done. You’d need to ask a whole new junior to explain it to you later (if you were relying on AI) At the recent startup I was at, the CEO (who was also an engineer) basically relied on chatgpt to make the whole app. It was a complicated and f…

Their metric doesn’t include old poor quality code that was recently replaced.

It only looks for brand new code that was immediately reverted/rewritten, and also looks for code that was copied from the internet vs. from within the repo (the former is probably non-idiomatic for the code base in question).

Re: Coding on Copilot: Data suggests downward pressure on code quality

#12

Post by hedora »

tussa wrote: post #6
naasking wrote: post #5

Possible downward pressure on code quality, but maybe it's partly because more people are getting involved in programming with coding assistance? That could be ok if there's more progress overall.

Would it? What does 1% reduction in code quality translate to in terms of delivery time and profits? I don't know. But I'm pretty sure 50% reduction results in project failure and company bankruptcy.

The study shows a doubling in the rate at which tech debt code is produced and checked into the repo.

Anecdotally, as a principal engineer, I’ve definitely noticed that new senior engineers on the team that say they are using chatgpt/copilot produce unprecedentedly bad code at unprecedented rates.

It takes me 2-3x longer to unwind such crap than it would for me to write it from scratch.

As we grow the team, this will definitely put us out of business unless we find a way to fix it.

Currently, we’re hoping the AI assisted engineers will get better at unborking code before merging it, but that’s a harder task than RTFM or going to stack overflow to copy-paste.

Re: Coding on Copilot: Data suggests downward pressure on code quality

#14

Post by jcutrell »

tussa wrote: post #6
naasking wrote: post #5

Possible downward pressure on code quality, but maybe it's partly because more people are getting involved in programming with coding assistance? That could be ok if there's more progress overall.

Would it? What does 1% reduction in code quality translate to in terms of delivery time and profits? I don't know. But I'm pretty sure 50% reduction results in project failure and company bankruptcy.

Kinda have to qualify "quality" by varied standards. Is it good enough for what the author needs? Then what matter is it that some measure of quality is not hit?

Re: Coding on Copilot: Data suggests downward pressure on code quality

#15

Post by AlwaysRock »

tussa wrote: post #6
naasking wrote: post #5

Possible downward pressure on code quality, but maybe it's partly because more people are getting involved in programming with coding assistance? That could be ok if there's more progress overall.

Would it? What does 1% reduction in code quality translate to in terms of delivery time and profits? I don't know. But I'm pretty sure 50% reduction results in project failure and company bankruptcy.

Depends a lot on the type of code. Front end code? Almost any website has some js/css/html errors on it and most of the time its not that big a deal. Backend bank routing code? Pretty important to get right.

From what I've seen, many early career engineers are the ones using code assist tools. Many early career engineers are often placed on lower stakes front end focused teams as well. This is mostly anecdotal data from non-traditional cs background engs that I know.

Re: Coding on Copilot: Data suggests downward pressure on code quality

#17

Post by evilantnie »

Copilot is a lossy compression algorithm when applied at scale, I would expect some degradation in code quality if not applied appropriately. It's still a useful tool, but just like image and audio compression, a human needs to give the final output some last looks to ensure an acceptable out come.

Re: Coding on Copilot: Data suggests downward pressure on code quality

#18

Post by gtirloni »

hackernoteng wrote: post #13

Gee really? You mean generated code by some ML model means people get lazier and just cut/paste hallucinated code? Who would have thought?

I doubt AI is making anyone lazier and less worried about code quality, if they weren't already.

Re: Coding on Copilot: Data suggests downward pressure on code quality

#19

Post by jerf »

hedora wrote: post #12
tussa wrote: post #6

Earlier quoted context omitted.

Would it? What does 1% reduction in code quality translate to in terms of delivery time and profits? I don't know. But I'm pretty sure 50% reduction results in project failure and company bankruptcy.

The study shows a doubling in the rate at which tech debt code is produced and checked into the repo. Anecdotally, as a principal engineer, I’ve definitely noticed that new senior engineers on the team that say they are using chatgpt/copilot produce unprecedentedly bad code at unprecedented rates. It takes me 2-3x longer to unwind such crap than it would for me to write it from scratch. As we grow the team, this will…

I don't have great English words for this, but the biggest concern for me with LLMs is that of all the text generation algorithms I've ever seen, they are just fantastic at producing output whose plausibility to the human mind greatly exceeds its actual quality, the difficulty of concretely measuring either of those values notwithstanding.

Note I'm not even strictly speaking criticizing the quality of the output per se. It is also a big jump over any previous technology and very impressive in its own way.

It is, nevertheless, quite dangerous because the jump in the human-perceived plausibility is much larger than the quality improvement.

Whereas earlier techs were obviously wrong to a human reader, in the case of code generation so obviously wrong that we never even considered using them, LLMs are extremely good at hiding the errors in the parts of the code that we are cognitively most inclined to overlook. This also has the effect of making it bizarrely difficult code to fix.

How it does this I do not know. A fascinating research question for some ambitious cognitive scientist. But the signal is very strong and I don't need to wait for a paper to come out to see it.

I do not think this is fundamental to AI. As I like to remind people, LLMs are not the whole of AI. They're just one technique, and one that partially for the very reason I discuss in this post, one I expect to eventually become a part of a larger system that can fix this problem at some higher level. I expect people to someday look back and laugh at us for thinking that LLMs could be used for all the things we think they can be used for. But the reasons they will be laughing are the very experience we're gathering now, and there's no skipping that phase.

Re: Coding on Copilot: Data suggests downward pressure on code quality

#20

Post by rabuse »

hedora wrote: post #12
tussa wrote: post #6

Earlier quoted context omitted.

Would it? What does 1% reduction in code quality translate to in terms of delivery time and profits? I don't know. But I'm pretty sure 50% reduction results in project failure and company bankruptcy.

The study shows a doubling in the rate at which tech debt code is produced and checked into the repo. Anecdotally, as a principal engineer, I’ve definitely noticed that new senior engineers on the team that say they are using chatgpt/copilot produce unprecedentedly bad code at unprecedented rates. It takes me 2-3x longer to unwind such crap than it would for me to write it from scratch. As we grow the team, this will…

I was going to ask, why aren't they running the code through unit tests if they're committing such shit code? Does it pass the tests, or is it just inefficient code?
Post reply on HN