Coding on Copilot: Data suggests downward pressure on code quality

Post by stcredzero » Tue, Jan 30, 2024, 1:06 AM UTC

dathinab wrote: post #39

stcredzero wrote: post #26

I think this is what's happening. Code production speed has increased. But the speed and efficiency of review and testing hasn't kept pace. Also, my current job limits the use of AI coding tools, because of the risk of our IP leaking out. To really be useful, there is a lot of context a programmer should be able to collate. Not all of it is going to be in the repo.

IMHO there is some form of crisis of "value production speed" for various programming things. And AI gives you fast code production speed so it seems to be a fix. But due to less structured code and code reuse it needs much more code for the same value produced. And it loves to produce the kind of bugs which are easy to pass reviews and tests (because this are the kind of bugs it most likely has as "correct" code in…

Post by stinos » Tue, Jan 30, 2024, 10:13 AM UTC

nzach wrote: post #45

>[...]"Churn" is the percentage of code that was pushed to the repo, then subsequently reverted, removed or updated within 2 weeks. This was a relatively infrequent outcome when developers authored all their own code -- only 3-4% of code was churned prior to 2023, although there is also a hint of the coming uptick in 2022, when Churn jumped 9%. 2022 was the first year Copilot was available in beta, and the year that…

    //This makes everything work somehow
    sleep(1)

Post by jprete » Tue, Jan 30, 2024, 7:22 PM UTC

naasking wrote: post #23

jprete wrote: post #8

Earlier quoted context omitted.

That's possible, but I don't think that would be a positive development either. Code isn't flexible or self-correcting, if you make bad code it will just break by surprise. If the only way someone can code is with LLM assistance then they won't even know what went wrong when the code breaks or takes forever or doesn't handle real corner cases. That said, I've done no LLM coding and don't have a feel for how it goes w…

> Code isn't flexible or self-correcting Code isn't, yes, but LLMs are . That's part of the key here, coding can in some sense now be done at the prompt level along with an acceptance test suite, so rather than coding solution and test suite, you can hypothetically just write the test suite and have the LLM generated the code part.

Post by naasking » Wed, Jan 31, 2024, 2:13 AM UTC

jprete wrote: post #55

naasking wrote: post #23

Earlier quoted context omitted.

> Code isn't flexible or self-correcting Code isn't, yes, but LLMs are . That's part of the key here, coding can in some sense now be done at the prompt level along with an acceptance test suite, so rather than coding solution and test suite, you can hypothetically just write the test suite and have the LLM generated the code part.

That doesn't help, coding newbies can't be expected to write comprehensive unit tests to keep an LLM's implementation on the rails, and the LLM flexibility doesn't stretch all the way into deployment unless it is the actual component being deployed.

Post by tomrod » Tue, Feb 06, 2024, 6:19 PM UTC

jerf wrote: post #19

hedora wrote: post #12

Earlier quoted context omitted.

The study shows a doubling in the rate at which tech debt code is produced and checked into the repo. Anecdotally, as a principal engineer, I’ve definitely noticed that new senior engineers on the team that say they are using chatgpt/copilot produce unprecedentedly bad code at unprecedented rates. It takes me 2-3x longer to unwind such crap than it would for me to write it from scratch. As we grow the team, this will…

I don't have great English words for this, but the biggest concern for me with LLMs is that of all the text generation algorithms I've ever seen, they are just fantastic at producing output whose plausibility to the human mind greatly exceeds its actual quality , the difficulty of concretely measuring either of those values notwithstanding. Note I'm not even strictly speaking criticizing the quality of the output per…