21–30 of 33 posts
"Developers still use Stack Overflow and other human sources of expertise" press X to doubt
"No, the developers do that."
"So you must take the answers to the person with the question?"
"Well, no... my LLM does that!"
"...what would you say you DO here?"
"I'm an answer website! What don't you people understand?!"
It's remarkable that academia hasn't produced rigorous studies with statistically significant samples, with hundreds of developers across languages and skill levels, to properly assess GenAI impact on productivity. The latest published research has a sample size of like...16 developers? We don't even need human subjects. GitHub contains a natural experiment: Millions of high-quality commits and PRs from before the Ge…
This has been the case in software development for a very long time, it is almost impossible to get reliable scientific data about these things because it's almost impossible to run a controlled experiment, and even if you did it's very difficult to decide what to measure and how to measure it. (It's easy to say "compare complexity and quality metrics" but how the fk are you going to define that? Does it really make…
I've cleaned up plenty of "almost right" code written by humans, even by me.
Earlier quoted context omitted.
This has been the case in software development for a very long time, it is almost impossible to get reliable scientific data about these things because it's almost impossible to run a controlled experiment, and even if you did it's very difficult to decide what to measure and how to measure it. (It's easy to say "compare complexity and quality metrics" but how the fk are you going to define that? Does it really make…
Your point cuts both ways: If measuring GenAI impact is as difficult as you say, then the bold claims of dramatic productivity gains are just as fragile. How, exactly, do nine out of ten CEOs know it works?
> One of the most surprising findings was a significant shift in developer preferences for AI compared to previous years, while most developers use AI, they like it less and trust it less this year I bet that is surprising, at least if you're astonishingly naive and believe everything a company tells you when they're trying to sell you something.
> One of the most surprising findings was a significant shift in developer preferences for AI compared to previous years, while most developers use AI, they like it less and trust it less this year I bet that is surprising, at least if you're astonishingly naive and believe everything a company tells you when they're trying to sell you something.
You mean if you're under 30 years old, like half of the people in the world? Or pick an age at which being "astonishingly naive" should be expected, since you just don't have the life experience yet. The proportion of people that age or younger may be surprising to you.
I've cleaned up plenty of "almost right" code written by humans, even by me.
The thing is a human can produce only so much code in a given time, with LLM(s) one can produce x times the same amount of code. Are you able to read, understand and clean up 5x or 10x as much code as 5 years ago ?
https://en.wikipedia.org/wiki/No_Silver_Bullet
Coding is 10% or 20% of the work, LLMs don’t radically improve throughput of software organizations
https://www.reddit.com/r/ExperiencedDevs/comments/1lwk503/st...
I see working with a LLM is like pair programming, if anything I end up wit better quality in the end because they see things in my blind spots, but there is no radical speed up.
We are past the "peak of inflated expectations" and entering the "trough of disillusionment" - which are the gartner hype-cycle phases for any new tech [1]. Just for reference (and whether you agree with it or not) the phases are 1) innovation, 2) peak of inflated expectations, 3) trough of disillusionment, 4) enlightenment and finally 5) plateau Of course, stack-overflow has every incentive to push this narrative, w…
The core issue with the Gartner hype cycle (beyond it not being a cycle!) is that it simply does not apply to "any new tech," tautologically it only applies to hype-driven tech with staying power. Tech which is not driven by hype (e.g. mRNA vaccines driven by urgent need) don't have a trough of disappointment, just a steady slope upwards. And tech which is driven entirely by hype (Theranos) stays in the trough of dis…
This disillusionment is evidenced by lower proportion of people seeking to take such injections now.
Earlier quoted context omitted.
The core issue with the Gartner hype cycle (beyond it not being a cycle!) is that it simply does not apply to "any new tech," tautologically it only applies to hype-driven tech with staying power. Tech which is not driven by hype (e.g. mRNA vaccines driven by urgent need) don't have a trough of disappointment, just a steady slope upwards. And tech which is driven entirely by hype (Theranos) stays in the trough of dis…
mRNA could be said to be in the trough of disillusionment -- didn't prevent COVID as initially promised, and has problems like causing heart damage in rare cases. This disillusionment is evidenced by lower proportion of people seeking to take such injections now.
Earlier quoted context omitted.
The core issue with the Gartner hype cycle (beyond it not being a cycle!) is that it simply does not apply to "any new tech," tautologically it only applies to hype-driven tech with staying power. Tech which is not driven by hype (e.g. mRNA vaccines driven by urgent need) don't have a trough of disappointment, just a steady slope upwards. And tech which is driven entirely by hype (Theranos) stays in the trough of dis…
mRNA could be said to be in the trough of disillusionment -- didn't prevent COVID as initially promised, and has problems like causing heart damage in rare cases. This disillusionment is evidenced by lower proportion of people seeking to take such injections now.
In Realityland, the COVID vaccine was an overwhelming success.