A Study of Microsoft's Early 2026 Rollout of Claude Code and GitHub Copilot CLI

Post by pqtyw » Tue, Jul 14, 2026, 6:58 AM UTC

sorokod wrote: post #30

onion2k wrote: post #28

Earlier quoted context omitted.

The problem that I'm finding in my own work on figuring out the impact of AI is that there's just no reliable way to connect things directly to AI usage. Most of the tooling does things like " The user used AI on the same day that they opened this PR, therefore we'll assume the AI was used to write the code in the PR. " In a mature AI-driven org that might be true, but in the rollout phase of an AI experiment it abso…

Completely disable the usage of AI for time period t. Compare the impact on whatever before t and during t.

Post by xamut » Tue, Jul 14, 2026, 7:11 AM UTC

strange_quark wrote: post #17

wilkystyle wrote: post #9

Earlier quoted context omitted.

I also think something along these lines is the correct answer. It can be hard to pin down an exact metric because once you start optimizing for a metric it tends to not be a good measure of the original thing anymore. But in general I think it comes down to some measure of feature velocity combined with a counter metric on support/maintenance burden. "Number of PRs merged" seems like "number of lines of code" wearin…

Feature velocity is another that's extremely easy to game. My company is trying this right now: instead of measuring PRs or lines of code, we are measuring number of customer facing features shipped. Well guess what? Everything is now a customer-facing feature. You did a big internal code refactor and data migration? Well guess what, that's a customer feature now because it unlocks future such and such. Deploy a new…

Post by pqtyw » Tue, Jul 14, 2026, 7:17 AM UTC

sorokod wrote: post #34

pqtyw wrote: post #31

Earlier quoted context omitted.

In any complex organization that might take several quarters if not more, though.

Possibly but this is not related to AI

Post by xamut » Tue, Jul 14, 2026, 7:21 AM UTC

jvuygbbkuurx wrote: post #36

xamut wrote: post #32

The number of merged PRs increased. But what does that actually tell us? There is no comparison of the complexity, or depth of those PRs with work previously done without agents. The authors themselves acknowledge that PR count rewards small, frequent PRs and that whether the additional output justifies its cost remains unanswered. We do not know whether end-to-end feature delivery became faster, whether the addition…

Especially in conditions where AI assisted PRs are tracked for performance. I had some extra usage on claude yesterday and put several ultra workflows to look for code cleanup opportunities. It made -40 tasks and PRs just like that. This is 10% of all PRs/commits in the project history, but the before and after is arguably meaningless.

Post by regularfry » Tue, Jul 14, 2026, 7:59 AM UTC

onion2k wrote: post #29

frostlynx wrote: post #11

Earlier quoted context omitted.

24% of increased productivity (yes, this is assuming of course that the “proxy” of merged PRs reflects productivity) is actually a pretty big deal. Given the salary of developers, this translates to tens of thousands of dollars per year, per developer. My guess is they used # of PRs as a measure as it’s easy to obtain, while other measures are hard, may be due to other factors, etc. FWIW I saw a similar number for my…

I've also seen an increase in merged PRs, but it coincided with developers opening smaller PRs. In other words, AI made devs break work down more so they opened more PRs for the same work. That's still good, because smaller increments are better, but there' no actual increase in coding productivity, and it means the context-switching burden from review work went up e.g developers slowed down in a different area.