Viewing profile — narush

[flagged]

We call this over-generalization out specifically in the "We do not provide evidence that:" table in the blog post and paper - I agree there are tasks these developers are likely s…

Hey, thanks for linking this! I'm a study author, and I greatly appreciate that this author dug into the appendix and provided feedback so that other folks can read it as well. A f…

Hey, thanks for digging into the details here! Copying a relevant comment ( https://news.ycombinator.com/item?id=44523638 ) from the other thread on the paper, in case it's help on…

Thanks for the feedback! I strongly agree this is not the only measure of developer productivity -- but it's certainly one of them. I think this measure as speaks very directly to …

Hey HN -- study author here! (See previous thread on the paper here [1].) I think this blog post is an interesting take on one specific factor that is likely contributing to slowdo…

Check out section AI increasing issue scope (C.2.3) in the paper -- https://metr.org/Early_2025_AI_Experienced_OS_Devs_Study.pdf We speak (the best we can) to changes in amount of …

How these results transfer to other settings is an excellent question. Previous literature would suggest speedup -- but I'd be excited to run a very similar methodology in those se…

Sorry, this is the first 8 issues per-developer!

Thank you!

We attempted to! We explore this more in the section Trading speed for ease (C.2.5) in the paper ( https://metr.org/Early_2025_AI_Experienced_OS_Devs_Study.pdf ). TLDR: mixed evide…

There's additional breakdown per-minute in the appendix -- see appendix E.4!

You can see a list of repositories with participating developers in the appendix! Section G.7. Paper is here: https://metr.org/Early_2025_AI_Experienced_OS_Devs_Study.pdf

You can see this analysis in the factor analysis of "Below-average use of AI tools" (C.2.7) in the paper [1], which we mark as an unclear effect. TLDR: over the first 8 issues, dev…

Thanks for the kind words!

The instructions given to developers was not just "implement with AI" - but rather that they could use AI if they deemed it would be helpful, but indeed did _not need to use AI if …

Honestly, this is a fair point -- and speaks the difficulty of figuring out the right baseline to measure against here! If we studied folks with _no_ AI experience, then we might u…

Yep, sorry, meant to post this somewhere but forgot in final-paper-polishing-sprint yesterday! We'll be releasing anonymized data and some basic analysis code to replicate core res…

> which feels like it is easier and hence faster. We explore this factor in section (C.2.5) - "Trading speed for ease" - in the paper [1]. It's labeled as a factor with an unclear …

The graphs are all matplotlib. The methodology figure is built in Figma! (Source: I'm a paper author :)).

Yeah, I'll note that this study does _not_ capture the entire OS dev workflow -- you're totally right that reviewing PRs is a big portion of the time that many maintainers spend on…

Qualitatively, we don't see a drop in PR quality in between AI-allowed and AI-disallowed conditions in the study; the devs who participate are generally excellent, know their repos…

Sounds great. Looking forward to hearing more detailed thoughts -- my emails in the paper :)

Noting that most of our power comes from the number of tasks that developers complete; it's 246 total completed issues in the course of this study -- developers do about 15 issues …

Hey Simon -- thanks for the detailed read of the paper - I'm a big fan of your OS projects! Noting a few important points here: 1. Some prior studies that find speedup do so with d…