Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

A Study of Microsoft's Early 2026 Rollout of Claude Code and GitHub Copilot CLI

arxiv.org

Post reply on HN
Print view View original

21–30 of 45 posts

Re: A Study of Microsoft's Early 2026 Rollout of Claude Code and GitHub Copilot CLI

#21

Post by btown »

frostlynx wrote: post #11
gojkoa wrote: post #2

The study unfortunately looks only at individual productivity, not any org gains, and the big claim in the PDF is that adopters "merge roughly 24% more pull requests" over a four month period. not exactly headline-making material. There's no data in the paper whether those 24% extra pull requests actually added anything more valuable or not.

24% of increased productivity (yes, this is assuming of course that the “proxy” of merged PRs reflects productivity) is actually a pretty big deal. Given the salary of developers, this translates to tens of thousands of dollars per year, per developer. My guess is they used # of PRs as a measure as it’s easy to obtain, while other measures are hard, may be due to other factors, etc. FWIW I saw a similar number for my…

Given that Microsoft's overhead to actually landing features has, from what I've heard, long been more about dodging the cross-org and inter-org "guns" famously depicted in [0] and [1] than coding time... a 24% increase in merged PRs is massive!

[0] https://static.ma.nu/publications/images/2013.07.12_new_york...

[1] https://newsletter.pragmaticengineer.com/i/138252015/drawing...

Re: A Study of Microsoft's Early 2026 Rollout of Claude Code and GitHub Copilot CLI

#22

Post by didibus »

jimsperry wrote: post #16
simonw wrote: post #4

Earlier quoted context omitted.

What kind of metric would you trust for measuring organization gains?

Microsoft itself has a system for measuring this which outlines a few example metrics: https://queue.acm.org/detail.cfm?id=3819080 There was a nice talk about this by one of the author's at this year's BUILD conference: https://build.microsoft.com/en-US/sessions/BRK210

Great read, thanks for sharing!

Re: A Study of Microsoft's Early 2026 Rollout of Claude Code and GitHub Copilot CLI

#23

Post by anshumankmr »

>281 billion tokens. Using the least expensive version of Claude Opus 4.6, which costs $5 for every million tokens, that one user alone could have cost Meta more than $1.4 million.

And that is based on the pricing that they sell a dollar for less than a dollar so the costs may be much much more.

Re: A Study of Microsoft's Early 2026 Rollout of Claude Code and GitHub Copilot CLI

#24

Post by pushplay »

frostlynx wrote: post #11
gojkoa wrote: post #2

The study unfortunately looks only at individual productivity, not any org gains, and the big claim in the PDF is that adopters "merge roughly 24% more pull requests" over a four month period. not exactly headline-making material. There's no data in the paper whether those 24% extra pull requests actually added anything more valuable or not.

24% of increased productivity (yes, this is assuming of course that the “proxy” of merged PRs reflects productivity) is actually a pretty big deal. Given the salary of developers, this translates to tens of thousands of dollars per year, per developer. My guess is they used # of PRs as a measure as it’s easy to obtain, while other measures are hard, may be due to other factors, etc. FWIW I saw a similar number for my…

> In my case this clearly doesn’t translate to as much value for the organization, or rather, it’s hard to say, as many of those PRs were things I wouldn’t even have done without AI support.

That's not necessarily bad. It could be a sign of effective prioritization. If you're good at working on the most important thing, and suddenly find yourself 24% more productive, what extra are you working on? The things that wouldn't have quite made the cut before.

Re: A Study of Microsoft's Early 2026 Rollout of Claude Code and GitHub Copilot CLI

#25

Post by meling »

Arainach wrote: post #20
frostlynx wrote: post #11

Earlier quoted context omitted.

24% of increased productivity (yes, this is assuming of course that the “proxy” of merged PRs reflects productivity) is actually a pretty big deal. Given the salary of developers, this translates to tens of thousands of dollars per year, per developer. My guess is they used # of PRs as a measure as it’s easy to obtain, while other measures are hard, may be due to other factors, etc. FWIW I saw a similar number for my…

24% PRs isn't 24% more productivity. Lines of code isn't productivity, and neither is CLs landed. What's the feature velocity of the team? How much time is being spent on rollbacks, outage responses, etc.? Here's a quick hack to triple your PRs landed: Land a PR, then land ANOTHER PR undoing that one when you realize it was full of bugs, then land the PR again once you realize management doesn't care about quality, t…

The 24% more PRs was merged, not «landed». Presumably after human code review (but I didn’t check the paper…) Not sure about this particular study, but I «think» my own productivity (coding wise) has improved by more than 30%. In many ways I wouldn’t even have started on many of the things I’ve completed these last six months because I would have viewed their effort to be insurmountable within my time budget as a professor (with all my other duties).

Re: A Study of Microsoft's Early 2026 Rollout of Claude Code and GitHub Copilot CLI

#26

Post by Traubenfuchs »

dnackoul wrote: post #10
simonw wrote: post #4

Earlier quoted context omitted.

What kind of metric would you trust for measuring organization gains?

We use a tool called Weave (I believe YC 25?) that analyzes PRs for "expert units of work" and shows lift from AI tools. My understanding is they have their own proprietary model that assesses the difficulty of each PR. I find the organization level view and pivots useful and aligned with intuitive expectations.

This sounds like voodoo.

Re: A Study of Microsoft's Early 2026 Rollout of Claude Code and GitHub Copilot CLI

#27

Post by simonreiff »

I really wish this study had chosen to focus on more useful endpoints like revenue, profitability, customer retention, conversion rate, etc....at a minimum, don't look at PRs merged without also looking at bugs in production code, number of incident reports, features shipped, debt log canceled, etc. Even if you only want to consider merged PRs, you should weigh the productivity of the reviewer and whether they are also 24% boosted or else it's just an accounting trick.

Re: A Study of Microsoft's Early 2026 Rollout of Claude Code and GitHub Copilot CLI

#28

Post by onion2k »

simonreiff wrote: post #27

I really wish this study had chosen to focus on more useful endpoints like revenue, profitability, customer retention, conversion rate, etc....at a minimum, don't look at PRs merged without also looking at bugs in production code, number of incident reports, features shipped, debt log canceled, etc. Even if you only want to consider merged PRs, you should weigh the productivity of the reviewer and whether they are al…

The problem that I'm finding in my own work on figuring out the impact of AI is that there's just no reliable way to connect things directly to AI usage. Most of the tooling does things like "The user used AI on the same day that they opened this PR, therefore we'll assume the AI was used to write the code in the PR." In a mature AI-driven org that might be true, but in the rollout phase of an AI experiment it absolutely isn't.

Until there's a good way to fix that gap in the data any measure of AI impact is going to be horribly flawed.

Re: A Study of Microsoft's Early 2026 Rollout of Claude Code and GitHub Copilot CLI

#29

Post by onion2k »

frostlynx wrote: post #11
gojkoa wrote: post #2

The study unfortunately looks only at individual productivity, not any org gains, and the big claim in the PDF is that adopters "merge roughly 24% more pull requests" over a four month period. not exactly headline-making material. There's no data in the paper whether those 24% extra pull requests actually added anything more valuable or not.

24% of increased productivity (yes, this is assuming of course that the “proxy” of merged PRs reflects productivity) is actually a pretty big deal. Given the salary of developers, this translates to tens of thousands of dollars per year, per developer. My guess is they used # of PRs as a measure as it’s easy to obtain, while other measures are hard, may be due to other factors, etc. FWIW I saw a similar number for my…

I've also seen an increase in merged PRs, but it coincided with developers opening smaller PRs. In other words, AI made devs break work down more so they opened more PRs for the same work. That's still good, because smaller increments are better, but there' no actual increase in coding productivity, and it means the context-switching burden from review work went up e.g developers slowed down in a different area.

Re: A Study of Microsoft's Early 2026 Rollout of Claude Code and GitHub Copilot CLI

#30

Post by sorokod »

onion2k wrote: post #28
simonreiff wrote: post #27

I really wish this study had chosen to focus on more useful endpoints like revenue, profitability, customer retention, conversion rate, etc....at a minimum, don't look at PRs merged without also looking at bugs in production code, number of incident reports, features shipped, debt log canceled, etc. Even if you only want to consider merged PRs, you should weigh the productivity of the reviewer and whether they are al…

The problem that I'm finding in my own work on figuring out the impact of AI is that there's just no reliable way to connect things directly to AI usage. Most of the tooling does things like " The user used AI on the same day that they opened this PR, therefore we'll assume the AI was used to write the code in the PR. " In a mature AI-driven org that might be true, but in the rollout phase of an AI experiment it abso…

Completely disable the usage of AI for time period t. Compare the impact on whatever before t and during t.
Post reply on HN