A Study of Microsoft's Early 2026 Rollout of Claude Code and GitHub Copilot CLI
1–10 of 45 posts
Re: A Study of Microsoft's Early 2026 Rollout of Claude Code and GitHub Copilot CLI
#2The study unfortunately looks only at individual productivity, not any org gains, and the big claim in the PDF is that adopters "merge roughly 24% more pull requests" over a four month period. not exactly headline-making material. There's no data in the paper whether those 24% extra pull requests actually added anything more valuable or not.
Re: A Study of Microsoft's Early 2026 Rollout of Claude Code and GitHub Copilot CLI
#3The study unfortunately looks only at individual productivity, not any org gains, and the big claim in the PDF is that adopters "merge roughly 24% more pull requests" over a four month period. not exactly headline-making material. There's no data in the paper whether those 24% extra pull requests actually added anything more valuable or not.
Highly doubt. Pace for merging requests have not improved and teams at MSFT are terrible at reviewing said PRs. Longer PRs and more frequent requests were clearly creating more friction.
Re: A Study of Microsoft's Early 2026 Rollout of Claude Code and GitHub Copilot CLI
#4The study unfortunately looks only at individual productivity, not any org gains, and the big claim in the PDF is that adopters "merge roughly 24% more pull requests" over a four month period. not exactly headline-making material. There's no data in the paper whether those 24% extra pull requests actually added anything more valuable or not.
What kind of metric would you trust for measuring organization gains?
Re: A Study of Microsoft's Early 2026 Rollout of Claude Code and GitHub Copilot CLI
#5The study unfortunately looks only at individual productivity, not any org gains, and the big claim in the PDF is that adopters "merge roughly 24% more pull requests" over a four month period. not exactly headline-making material. There's no data in the paper whether those 24% extra pull requests actually added anything more valuable or not.
Dependabot PR merges :rocket: :rocket:
Re: A Study of Microsoft's Early 2026 Rollout of Claude Code and GitHub Copilot CLI
#6The study unfortunately looks only at individual productivity, not any org gains, and the big claim in the PDF is that adopters "merge roughly 24% more pull requests" over a four month period. not exactly headline-making material. There's no data in the paper whether those 24% extra pull requests actually added anything more valuable or not.
What kind of metric would you trust for measuring organization gains?
It'll probably take a really good product built by a profitable company evangelizing an AI workflow with reproducible examples dating back a few years.
Re: A Study of Microsoft's Early 2026 Rollout of Claude Code and GitHub Copilot CLI
#7The study unfortunately looks only at individual productivity, not any org gains, and the big claim in the PDF is that adopters "merge roughly 24% more pull requests" over a four month period. not exactly headline-making material. There's no data in the paper whether those 24% extra pull requests actually added anything more valuable or not.
What kind of metric would you trust for measuring organization gains?
I think number of features released to customers (not behind a feature flag or still being rolled out, but fully rolled out). And number of bug fixes (only those reported by customers).
Also just in general, customer satisfaction, acquisition, conversion, retention, etc.
Number of completed org-level roadmap items, org-level goals achievement rate, and so on.
I also think a good one would be seeing an increase in meeting estimation, like if project was estimated to take X days with Y devs, does the use of AI increased how often you met or beat those estimates in actual time/dev effort?
And you'd want to compare that against prior years, where no AI was used, within the same org, or try going 1 quarter without AI and another with and compare quarter to quarter.
Re: A Study of Microsoft's Early 2026 Rollout of Claude Code and GitHub Copilot CLI
#8The study unfortunately looks only at individual productivity, not any org gains, and the big claim in the PDF is that adopters "merge roughly 24% more pull requests" over a four month period. not exactly headline-making material. There's no data in the paper whether those 24% extra pull requests actually added anything more valuable or not.
What kind of metric would you trust for measuring organization gains?
Engineering headcount.
Re: A Study of Microsoft's Early 2026 Rollout of Claude Code and GitHub Copilot CLI
#9Earlier quoted context omitted.
What kind of metric would you trust for measuring organization gains?
I think number of features released to customers (not behind a feature flag or still being rolled out, but fully rolled out). And number of bug fixes (only those reported by customers). Also just in general, customer satisfaction, acquisition, conversion, retention, etc. Number of completed org-level roadmap items, org-level goals achievement rate, and so on. I also think a good one would be seeing an increase in mee…
I also think something along these lines is the correct answer. It can be hard to pin down an exact metric because once you start optimizing for a metric it tends to not be a good measure of the original thing anymore. But in general I think it comes down to some measure of feature velocity combined with a counter metric on support/maintenance burden.
"Number of PRs merged" seems like "number of lines of code" wearing a trenchcoat, and I thought we all agreed back in the 90s that number of lines of code was a terrible measure of software productivity...
Re: A Study of Microsoft's Early 2026 Rollout of Claude Code and GitHub Copilot CLI
#10The study unfortunately looks only at individual productivity, not any org gains, and the big claim in the PDF is that adopters "merge roughly 24% more pull requests" over a four month period. not exactly headline-making material. There's no data in the paper whether those 24% extra pull requests actually added anything more valuable or not.
What kind of metric would you trust for measuring organization gains?
We use a tool called Weave (I believe YC 25?) that analyzes PRs for "expert units of work" and shows lift from AI tools. My understanding is they have their own proprietary model that assesses the difficulty of each PR. I find the organization level view and pivots useful and aligned with intuitive expectations.