Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

A Study of Microsoft's Early 2026 Rollout of Claude Code and GitHub Copilot CLI

arxiv.org

Post reply on HN
Print view View original

41–45 of 45 posts

Re: A Study of Microsoft's Early 2026 Rollout of Claude Code and GitHub Copilot CLI

#41

Post by dzonga »

simonreiff wrote: post #27

I really wish this study had chosen to focus on more useful endpoints like revenue, profitability, customer retention, conversion rate, etc....at a minimum, don't look at PRs merged without also looking at bugs in production code, number of incident reports, features shipped, debt log canceled, etc. Even if you only want to consider merged PRs, you should weigh the productivity of the reviewer and whether they are al…

negative ROI - when you start using actual business metrics.

it's in everyone's best interest to keep those numbers covered.

Re: A Study of Microsoft's Early 2026 Rollout of Claude Code and GitHub Copilot CLI

#42

Post by saghm »

onion2k wrote: post #28
simonreiff wrote: post #27

I really wish this study had chosen to focus on more useful endpoints like revenue, profitability, customer retention, conversion rate, etc....at a minimum, don't look at PRs merged without also looking at bugs in production code, number of incident reports, features shipped, debt log canceled, etc. Even if you only want to consider merged PRs, you should weigh the productivity of the reviewer and whether they are al…

The problem that I'm finding in my own work on figuring out the impact of AI is that there's just no reliable way to connect things directly to AI usage. Most of the tooling does things like " The user used AI on the same day that they opened this PR, therefore we'll assume the AI was used to write the code in the PR. " In a mature AI-driven org that might be true, but in the rollout phase of an AI experiment it abso…

It's wild how we need to rediscover Goodhart's Law every time we have a new thing we think might increase productivity.

Re: A Study of Microsoft's Early 2026 Rollout of Claude Code and GitHub Copilot CLI

#43

Post by gojkoa »

regularfry wrote: post #39
onion2k wrote: post #29

Earlier quoted context omitted.

I've also seen an increase in merged PRs, but it coincided with developers opening smaller PRs. In other words, AI made devs break work down more so they opened more PRs for the same work. That's still good, because smaller increments are better, but there' no actual increase in coding productivity, and it means the context-switching burden from review work went up e.g developers slowed down in a different area.

I'm yet to be convinced that number of PRs as a productivity metric is any less flawed than counting lines of code. I can believe - easily - that there's a real uplift here, but attaching any meaning to the 24% number at all is a massive overstep.

PRs measure effort, not outcome, so it’s a measure that their developers+ai spent 24% more effort than just devs previously (assuming the average work that went into a PR didn’t change, which is also a big assumption. maybe those PRs were smaller or more complex on average than before). so the uplift isn’t really tracked here. time to market, average cycle time per change, number of incidents per ticket released etc would be more meaningful metrics for uplift

Re: A Study of Microsoft's Early 2026 Rollout of Claude Code and GitHub Copilot CLI

#44

Post by gojkoa »

simonw wrote: post #4
gojkoa wrote: post #2

The study unfortunately looks only at individual productivity, not any org gains, and the big claim in the PDF is that adopters "merge roughly 24% more pull requests" over a four month period. not exactly headline-making material. There's no data in the paper whether those 24% extra pull requests actually added anything more valuable or not.

What kind of metric would you trust for measuring organization gains?

ultimately product success metrics such as market share and revenue, but those are lagging and might not be measurable over a 4 month period (the original study duration). beyond that, behaviour changes for users (is the product more useful?). those can be measured more frequently and can be compared to the previous period. if not that, than delivery output metrics such as cycle time for development (are we delivering faster?) paired with failure demand (% of tickets we thought were delivered but required rework/fixing). PRs are tied to effort/delivery process inputs, not outputs.

Re: A Study of Microsoft's Early 2026 Rollout of Claude Code and GitHub Copilot CLI

#45

Post by aioproductos »

simonreiff wrote: post #27

I really wish this study had chosen to focus on more useful endpoints like revenue, profitability, customer retention, conversion rate, etc....at a minimum, don't look at PRs merged without also looking at bugs in production code, number of incident reports, features shipped, debt log canceled, etc. Even if you only want to consider merged PRs, you should weigh the productivity of the reviewer and whether they are al…

[flagged]
Post reply on HN