Coding on Copilot: Data suggests downward pressure on code quality
1–10 of 60 posts
Re: Coding on Copilot: Data suggests downward pressure on code quality
#2Discussed previously: https://news.ycombinator.com/item?id=39168105
Re: Coding on Copilot: Data suggests downward pressure on code quality
#3"GitClear analyzed approximately 153 million changed lines of code, authored between January 2020 and December 2023 [A1]. This is the largest known database of highly structured code change data that has been used to evaluate code quality differences [A2]. We find disconcerting trends for maintainability. Code churn -- the percentage of lines that are reverted or updated less than two weeks after being authored -- is projected to double in 2024 compared to its 2021, pre-AI baseline. We further find that the percentage of "added code" and "copy/pasted code" is increasing in proportion to “updated,” “deleted,” and “moved” code. In this regard, code generated during 2023 more resembles an itinerant contributor, prone to violate the DRY-ness of the repos visited. "
If the cost of updating code has dropped due to AI, is code churn still a good proxy for low code quality?
Re: Coding on Copilot: Data suggests downward pressure on code quality
#4"GitClear analyzed approximately 153 million changed lines of code, authored between January 2020 and December 2023 [A1]. This is the largest known database of highly structured code change data that has been used to evaluate code quality differences [A2]. We find disconcerting trends for maintainability. Code churn -- the percentage of lines that are reverted or updated less than two weeks after being authored -- is…
Indeed - if anything in my organisation AI has lead to developers replacing poor quality existing code more quickly because what was quite a time consuming process before can be sped up tremendously by using language models.
Re: Coding on Copilot: Data suggests downward pressure on code quality
#5Possible downward pressure on code quality, but maybe it's partly because more people are getting involved in programming with coding assistance? That could be ok if there's more progress overall.
Re: Coding on Copilot: Data suggests downward pressure on code quality
#6Possible downward pressure on code quality, but maybe it's partly because more people are getting involved in programming with coding assistance? That could be ok if there's more progress overall.
Would it? What does 1% reduction in code quality translate to in terms of delivery time and profits?
I don't know. But I'm pretty sure 50% reduction results in project failure and company bankruptcy.
Re: Coding on Copilot: Data suggests downward pressure on code quality
#7"GitClear analyzed approximately 153 million changed lines of code, authored between January 2020 and December 2023 [A1]. This is the largest known database of highly structured code change data that has been used to evaluate code quality differences [A2]. We find disconcerting trends for maintainability. Code churn -- the percentage of lines that are reverted or updated less than two weeks after being authored -- is…
Indeed - if anything in my organisation AI has lead to developers replacing poor quality existing code more quickly because what was quite a time consuming process before can be sped up tremendously by using language models.
Replacing poor code, with poor code that nobody really understands isn’t good either.
Imagine hiring a junior to write your code, except they’d just stop existing when they were done.
You’d need to ask a whole new junior to explain it to you later (if you were relying on AI)
At the recent startup I was at, the CEO (who was also an engineer) basically relied on chatgpt to make the whole app.
It was a complicated and fragile codebase. The docstrings, which were also AI generated, were frequently out of date if not just wrong.
We were in the midst of rewriting it before we went bust.
Re: Coding on Copilot: Data suggests downward pressure on code quality
#8Possible downward pressure on code quality, but maybe it's partly because more people are getting involved in programming with coding assistance? That could be ok if there's more progress overall.
That's possible, but I don't think that would be a positive development either. Code isn't flexible or self-correcting, if you make bad code it will just break by surprise. If the only way someone can code is with LLM assistance then they won't even know what went wrong when the code breaks or takes forever or doesn't handle real corner cases.
That said, I've done no LLM coding and don't have a feel for how it goes wrong.
Re: Coding on Copilot: Data suggests downward pressure on code quality
#9Possible downward pressure on code quality, but maybe it's partly because more people are getting involved in programming with coding assistance? That could be ok if there's more progress overall.
Would it? What does 1% reduction in code quality translate to in terms of delivery time and profits? I don't know. But I'm pretty sure 50% reduction results in project failure and company bankruptcy.
Depends where we're seeing this.
I worked as a high school bureaucrat for several years, and through some simple scripts, I was able to make some tedious data entry tasks vastly faster. It was ugly, hacked together code, with lots of hardcoded values I had to update each semester, but it worked and was way better than doing it all by hand. Low quality code is worse than good quality code, but often better than manual labor.
I want my web browser and my bank to use high quality code, but I want semi-technical people who would otherwise do everything manually to be able to automate tedious portions of their jobs.
(The caveat, of course, being that a buggy script can screw up the thing the thing it's automating must faster than a manual process as well.)
Re: Coding on Copilot: Data suggests downward pressure on code quality
#10Earlier quoted context omitted.
Indeed - if anything in my organisation AI has lead to developers replacing poor quality existing code more quickly because what was quite a time consuming process before can be sped up tremendously by using language models.
Replacing poor code, with poor code that nobody really understands isn’t good either. Imagine hiring a junior to write your code, except they’d just stop existing when they were done. You’d need to ask a whole new junior to explain it to you later (if you were relying on AI) At the recent startup I was at, the CEO (who was also an engineer) basically relied on chatgpt to make the whole app. It was a complicated and f…
Well if you need it touch it again, you hope for an even better AI.