Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Preliminary data from a longitudinal AI impact study

newsletter.getdx.com

Post reply on HN
Print view View original

11–20 of 42 posts

Re: Preliminary data from a longitudinal AI impact study

#11

Post by felipeerias »

rybosworld wrote: post #8

> Planning, alignment, scoping, code review, and handoffs—the human parts of the SDLC—remain largely untouched Seems likely that process is holding things back. Planning has always been a "best-guess". There's lots you can't account for until you start a task. Code review mostly exists because the cost of doing something wrong was high (because human coding is slow). If you can code faster, you can replace bad code f…

Planning might end up being more reliable thanks to coding agents: if you want to estimate how long a task would take, just send an agent to do it.

If the agent comes back in a few minutes with a tiny fix, it is probably a small task.

If the agent produces a large, convoluted solution that would need careful review, it is at least a medium task.

And if the agent gets stuck, runs into architectural constraints, etc. then it is definitely a hard task.

Re: Preliminary data from a longitudinal AI impact study

#12

Post by SiempreViernes »

eucyclos wrote: post #9
arisAlexis wrote: post #4

because the human may be the bottleneck soon

It might be more accurate to say humans will only work at the bottlenecks soon, unless I've misunderstood the vector of your commentary.

A lot of AI-boosting commentary does speak in terms where there are hardly any humans left in their dream world, so it makes sense to ask if that's what they mean!

Re: Preliminary data from a longitudinal AI impact study

#14

Post by esseph »

enraged_camel wrote: post #7

>> November 2024 through February 2026 Yeah, listen... I'm glad these types of studies are being conducted. I'll say this though: the difference between pre- and post-Opus 4.5 has been night and day for me. From August 2025 through November 2025 I led a complex project at work where I used Sonnet 4.5 heavily. It was very helpful, but my total productivity gains were around 10-15%, which is pretty much what the study…

I swear people say this with every single model and release version, without fail.

Re: Preliminary data from a longitudinal AI impact study

#15

Post by jackschultz »

enraged_camel wrote: post #7

>> November 2024 through February 2026 Yeah, listen... I'm glad these types of studies are being conducted. I'll say this though: the difference between pre- and post-Opus 4.5 has been night and day for me. From August 2025 through November 2025 I led a complex project at work where I used Sonnet 4.5 heavily. It was very helpful, but my total productivity gains were around 10-15%, which is pretty much what the study…

Very much agree. Gave a presentation on AI to a group earlier this week and I spent a third of the time talking about the Opus 4.5 inflection point in AI history. First time using that model the day it was released it was so clear that it knew what it was doing at a different level. People still jump around to different models or tools or time frames when talking about AI and usefulness, but those have no meaning if they’re not using the Opus 4.5 and 4.6 models and anthropic harnesses of Claude code or cowork.

I’m interested in the studies along with the history of AI and if they’re going to realize that was the point when things changed, because for us devs, that was the moment.

Re: Preliminary data from a longitudinal AI impact study

#16

Post by Insanity »

naasking wrote: post #13

Sounds reasonable, but gains will go up. There is a ceiling somewhere, but we don't know where it is.

Yup, and the ceiling could be at 11% or at 50%. But my bet is closer to a lower-range ceiling than an upper-range. Model's are no longer revolutionary, they are evolutionary, and the evolution and per model-version difference is narrowing each release.

Re: Preliminary data from a longitudinal AI impact study

#17

Post by slopinthebag »

enraged_camel wrote: post #7

>> November 2024 through February 2026 Yeah, listen... I'm glad these types of studies are being conducted. I'll say this though: the difference between pre- and post-Opus 4.5 has been night and day for me. From August 2025 through November 2025 I led a complex project at work where I used Sonnet 4.5 heavily. It was very helpful, but my total productivity gains were around 10-15%, which is pretty much what the study…

> It was very helpful, but my total productivity gains were around 10-15%, which is pretty much what the study found. Once Opus came out in November though, it was like someone flipped a switch. It was much more capable at autonomous work and required way less hand-holding, intervention or course-correction. 4.6 has been even better.

How did you track these gains?

Re: Preliminary data from a longitudinal AI impact study

#18

Post by jongjong »

As I've said before, AI is a force multiplier. A 10x developer is now a 100x developer and a -10x developer (complexity maker/value destroyer) is now a -100x developer.

I can understand why a lot of companies are cutting junior roles. What AI does is it automates most of the stuff that juniors are good at (coding fast) but not much of the stuff that the seniors are good at.

That said, I've worked with some juniors who managed to navigate; they do this by focusing on higher order thinking and developing a sense of what's important by interacting with senior engineers. Unfortunately, it raises the talent bar for juniors; they have to become more intelligent; not in a puzzle-solving way, but in a more architectural big-picture sort of way; almost like entrepreneurial thinking but more detailed/complex.

LLMs don't have a worldview; this means that they miss a lot of inconsistencies and logical contradictions. Also, most critically, LLMs don't know what's important (at least not accurately enough) so they can't prioritize effectively and they make a lot of bad decisions.

It's kind of interesting for me because a lot of the areas where I had a contrarian opinion in the field of software development, I now see LLMs getting trapped into those and getting bad results. It's like all my contrarian opinions became much more valuable.

Re: Preliminary data from a longitudinal AI impact study

#19

Post by SirensOfTitan »

This reads as incredibly damning to me. PR throughput should be a metric that is very supportive of the AI productivity narrative, but the effect is marginal.

Before everyone gets at me: smoking cigarettes increases your risk of lung cancer by 15-30x. Effect size matters. As does margin of error: what is the margin of error? This "increase" could easily be within noise.

PR throughput is also not a metric I would ever use to determine developer productivity for a paradigm shifting technology. I would only ever use it to compare like-to-like to find trailheads: is a team or person suddenly way more or less productive? The primary endpoint for software production is serving your customer or your mission, and PR throughput can't tell you whether any of that got better. It also cannot tell you the cost of your prior work: the increase in PR throughput could be more PRs to fix issues introduced by LLM-assisted work.

Re: Preliminary data from a longitudinal AI impact study

#20

Post by nubg »

0xbadc0de5 wrote: post #6

Fair assessment. And worth noting that in a sane world, a broad 10% productivity improvement across industry would be a once-in-a-lifetime, headline-making story, not a disappointment.

Agreed, but if that came at a cost of 1 trillion dollars of debt and investments, it might be a disappointment again.

Note that I am bullish on AI coding in general, just trying to contextualize your statement.

Post reply on HN