Stack Overflow data reveals the hidden productivity tax of almost right AI code
1–10 of 33 posts
Re: Stack Overflow data reveals the hidden productivity tax of almost right AI code
#2It's remarkable that academia hasn't produced rigorous studies with statistically significant samples, with hundreds of developers across languages and skill levels, to properly assess GenAI impact on productivity. The latest published research has a sample size of like...16 developers?
We don't even need human subjects. GitHub contains a natural experiment: Millions of high-quality commits and PRs from before the GenAI explosion (pre-Jan 2021). Compare velocity, complexity, and quality metrics before and after, the data is sitting there, waiting to be analyzed.
Instead, we're drowning in anecdotes while petabytes of empirical evidence remain untouched.
Re: Stack Overflow data reveals the hidden productivity tax of almost right AI code
#3We are past the "peak of inflated expectations" and entering the "trough of disillusionment" - which are the gartner hype-cycle phases for any new tech [1]. Just for reference (and whether you agree with it or not) the phases are 1) innovation, 2) peak of inflated expectations, 3) trough of disillusionment, 4) enlightenment and finally 5) plateau
Of course, stack-overflow has every incentive to push this narrative, which the LLM makers will counter vigorously. The end-consumers (developers) will be the jury.
[1] https://www.gartner.com/en/articles/hype-cycle-for-artificia...
Re: Stack Overflow data reveals the hidden productivity tax of almost right AI code
#4As if humans always produce high quality always right working code without tech debt. The article is full of AI cliche echos.
Re: Stack Overflow data reveals the hidden productivity tax of almost right AI code
#5It reminds me of “Debugging is twice as hard as writing the code in the first place” (Brian Kernighan).
If you use AI to write the code, did you ever make it to the 1x level to begin with? How can you be 2x smarter to debug it if you didn’t reach the 1x level initially?
Re: Stack Overflow data reveals the hidden productivity tax of almost right AI code
#6It reminds me of “Debugging is twice as hard as writing the code in the first place” (Brian Kernighan). If you use AI to write the code, did you ever make it to the 1x level to begin with? How can you be 2x smarter to debug it if you didn’t reach the 1x level initially?
> How can you be 2x smarter to debug it if you didn’t reach the 1x level initially?
Apparently the solution will be 10x prompters.
Re: Stack Overflow data reveals the hidden productivity tax of almost right AI code
#7It's remarkable that academia hasn't produced rigorous studies with statistically significant samples, with hundreds of developers across languages and skill levels, to properly assess GenAI impact on productivity. The latest published research has a sample size of like...16 developers? We don't even need human subjects. GitHub contains a natural experiment: Millions of high-quality commits and PRs from before the Ge…
As in every bubble, its the incentives.
Everyone is selling AI or being sold AI.
Re: Stack Overflow data reveals the hidden productivity tax of almost right AI code
#8"Developers still use Stack Overflow and other human sources of expertise"
press X to doubt
Re: Stack Overflow data reveals the hidden productivity tax of almost right AI code
#9I've cleaned up plenty of "almost right" code written by humans, even by me.
Re: Stack Overflow data reveals the hidden productivity tax of almost right AI code
#10It reminds me of “Debugging is twice as hard as writing the code in the first place” (Brian Kernighan). If you use AI to write the code, did you ever make it to the 1x level to begin with? How can you be 2x smarter to debug it if you didn’t reach the 1x level initially?
> How can you be 2x smarter to debug it if you didn’t reach the 1x level initially? Apparently the solution will be 10x prompters.
No, 10x reviewers.