Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Coding on Copilot: Data suggests downward pressure on code quality

gitclear.com

Post reply on HN
Print view View original

41–50 of 60 posts

Re: Coding on Copilot: Data suggests downward pressure on code quality

#41

Post by lostmsu »

datadrivenangel wrote: post #3

"GitClear analyzed approximately 153 million changed lines of code, authored between January 2020 and December 2023 [A1]. This is the largest known database of highly structured code change data that has been used to evaluate code quality differences [A2]. We find disconcerting trends for maintainability. Code churn -- the percentage of lines that are reverted or updated less than two weeks after being authored -- is…

This could also be an indication of young codebases or more frequent commits.

Basically, the AI correlation is baseless.

Re: Coding on Copilot: Data suggests downward pressure on code quality

#42

Post by naasking »

nicup12345689 wrote: post #25
naasking wrote: post #5

Possible downward pressure on code quality, but maybe it's partly because more people are getting involved in programming with coding assistance? That could be ok if there's more progress overall.

What is your definition of progress?

Greater quantity of useful open source code.

Re: Coding on Copilot: Data suggests downward pressure on code quality

#44

Post by flir »

cjbgkagh wrote: post #22
gtirloni wrote: post #18

Earlier quoted context omitted.

I doubt AI is making anyone lazier and less worried about code quality, if they weren't already.

People can learn things even if they would rather not. Allowing them to take shortcuts will diminish their learning opportunities.

I'm not ashamed to admit I'm learning a lot from the chatbots. And it seems to stick better, as compared to reading a manual.

Re: Coding on Copilot: Data suggests downward pressure on code quality

#45

Post by nzach »

>[...]"Churn" is the percentage of code that was pushed to the repo, then subsequently reverted, removed or updated within 2 weeks. This was a relatively infrequent outcome when developers authored all their own code -- only 3-4% of code was churned prior to 2023, although there is also a hint of the coming uptick in 2022, when Churn jumped 9%. 2022 was the first year Copilot was available in beta, and the year that ChatGPT became available.

I think this is a bad take on this issue. It implies that if either chatgpt or the devs were just "smarter" this would not happen. This also implies that churn is a bad thing. And I don't think neither of these are true.

The real problem is that debugging is twice as hard as programming and now for the first time we got to the point that people can create software that "works" but they cannot fully grasp why exactly it works.

Of course you can go with the easy answer (these people are just lazy/dumb and should not be in this field) but this doesn't help. But the reality is that we don't really have anything better than printf and gdb to understand what the is happening inside our computers.

Hopefully now we will have more incentives to create better tools to help our understanding of what happens under the hood.

Re: Coding on Copilot: Data suggests downward pressure on code quality

#46

Post by squigz »

nzach wrote: post #45

>[...]"Churn" is the percentage of code that was pushed to the repo, then subsequently reverted, removed or updated within 2 weeks. This was a relatively infrequent outcome when developers authored all their own code -- only 3-4% of code was churned prior to 2023, although there is also a hint of the coming uptick in 2022, when Churn jumped 9%. 2022 was the first year Copilot was available in beta, and the year that…

> we got to the point that people can create software that "works" but they cannot fully grasp why exactly it works.

It seems to me that this is the real problem. The difference between a sub-par programmer and a competent one is that the latter can create code that they understand and can therefore more easily debug. Being able to create code that "works", but you then have to spend hours/days trying to grok in order to debug it, seems to me to be more harmful than productive.

Re: Coding on Copilot: Data suggests downward pressure on code quality

#47

Post by pinkmuffinere »

jfisher4024 wrote: post #30

I used to use it mainly to write unit tests, which is a nice use case because it’s obvious if it gets it wrong. (The test fails.) One time I had to use the Go protobuff reflection API, and Copilot was actual crucial because the docs are so bad. Can’t use it in my current role and I feel less productive.

This seems to me like the perfect use case, I’ll have to try using it for unit tests. Are you ever concerned when the AI-written unit test passes that it might be passing incorrectly? Either for the wrong reason, or when it should fail?

Re: Coding on Copilot: Data suggests downward pressure on code quality

#48

Post by gtirloni »

flir wrote: post #44
cjbgkagh wrote: post #22

Earlier quoted context omitted.

People can learn things even if they would rather not. Allowing them to take shortcuts will diminish their learning opportunities.

I'm not ashamed to admit I'm learning a lot from the chatbots. And it seems to stick better, as compared to reading a manual.

Same here. Instead of me conforming my thoughts to some document, the chatbot works together with me and my questions. I find them very useful form learning, so much I usually start a learning session first before I ever try to search something on Google.

Back to code solutions, I never ever use them if I don't understand ALL that is happening in that code. I know the bot will ignore many corner cases or that it doesn't know some subtle implications about the context where I'll deploy the code.

I think someone that doesn't care about code quality was already copying and pasting code from StackOverflow and fisting the keyboard until it works then moving to the next ticket. I really don't think AI is making things worse in that regard.

Re: Coding on Copilot: Data suggests downward pressure on code quality

#49

Post by JohnFen »

ctoth wrote: post #34
hackernoteng wrote: post #13

Gee really? You mean generated code by some ML model means people get lazier and just cut/paste hallucinated code? Who would have thought?

Actually if you read the study (I didn't want to give them my email either) you'll find that they can't tell generated code apart from any other code, and are just saying that code quality went down (per their metrics) in 2023. They assume it is because of Copilot. They do not demonstrate this.

> They assume it is because of Copilot. They do not demonstrate this.

True.

It does, however, correspond with my observations of devs at my workplace. Those who have been using copilot (and the like) have had fairly obvious drops in the quality of their work.

Post reply on HN