> We pay developers $150/hr as compensation for their participation in the study. Can someone point me to these 300k/yr jobs?
This is more like contractor pay rates -- not salaried. Which is appropriate to the nature of the work here.
Measuring the impact of AI on experienced open-source developer productivity
351–360 of 501 posts
Re: Measuring the impact of AI on experienced open-source developer productivity
#352I really admire stories like this. Reaching $1M ARR without any funding is rare and feels real. It shows what building something truly takes. Late nights, tough moments, losing users. It's not about big bursts of growth but staying consistent, solving real problems, and growing revenue little by little. There's a lot to learn from that.
Wrong thread
Re: Measuring the impact of AI on experienced open-source developer productivity
#353Earlier quoted context omitted.
I had a bad time with Cursor. I use Claude Code inside of VS: Code. You don't necessarily need Max, but you can spend a lot of money very quickly on API tokens, so I'd recommend to anyone trying, start with the $20/month one, no need to spend a ton of money just to try something out. There is a skill gap, like, I think of it like vim: at first it slows you down, but then as you learn it, you end up speeding up. So yo…
Worth noting for the folks asking: there's an official Claude Code extension for VS Code now [0]. I haven't tried it personally, but that's mostly because I mainly use the terminal and vim. [0]: https://marketplace.visualstudio.com/items?itemName=anthropi...
Yes, it’s not necessary but it is convenient for viewing diffs in Code’s diff view. The terminal is a fine way to interact with it though.
Re: Measuring the impact of AI on experienced open-source developer productivity
#354Something I don’t see mentioned that’s been helpful to me is having an agent add strict type safety to my typescript. I avoid the use of “any” type and berating an agent to “make it work” really opens my eyes and forces me to learn how advanced typescript can be leveraged. I feel that creating complex types that make my code simpler, makes autocomplete work(!), is a great tradeoff in some meta dimension of software dev.
Re: Measuring the impact of AI on experienced open-source developer productivity
#355would you be worse without it? now prepare to pay $1000+/month for chatgpt in a few years when dust settles.
Re: Measuring the impact of AI on experienced open-source developer productivity
#356I would say AI has very different impacts on individual coding styles.
Re: Measuring the impact of AI on experienced open-source developer productivity
#357Here's the full paper, which has a lot of details missing from the summary linked above: https://metr.org/Early_2025_AI_Experienced_OS_Devs_Study.pdf My personal theory is that getting a significant productivity boost from LLM assistance and AI tools has a much steeper learning curve than most people expect. This study had 16 participants, with a mix of previous exposure to AI tools - 56% of them had never used Curso…
> My intuition here is that this study mainly demonstrated that the learning curve on AI-assisted development is high enough that asking developers to bake it into their existing workflows reduces their performance while they climb that learing curve. Definitely. Effective LLM usage is not as straightforward as people believe. Two big things I see a lot of developers do when they share chats: 1. Talk to the LLM like…
> But I can assure you that "pandas count unique values column 'Foo'" is just as effective an LLM prompt as "Using pandas, how do I get the count of unique values in the column named 'Foo'?"
While the results are going to be similar, typing a question in full can help you think about it yourself too, as if the LLM is a rubber duck that can respond back.
I've found myself adjusting and rewriting prompts during the process of writing them before i ask the LLM anything because as i was writing the prompt i was thinking about the problem simultaneously.
Of course for simple queries like "write me a function in C that calculates the length of a 3d vector using vec3 for type" you can write it like "c function vec3 length 3d" or something like that instead and the LLM will give more or less the same response (tried it with Devstral).
But TBH to me that sounds like programmers using Vim claiming they're more productive than users of other editors because they have to use less keystrokes.
Re: Measuring the impact of AI on experienced open-source developer productivity
#358As someone has been doing hardcore genai for 2+ years, my experience has been, and what we advise internally: * 3 weeks to transition from ai pairing to AI Delegation to ai multitasking. So work gains are mostly week 3+. That's 120+ hours in, as someone pretty senior here. * Speedup is the wrong metric. Think throughput, not latency. Some finite amount of work might take longer, but the volume of work should go up be…
Have you actually measured this? Because one of the big takeaways from this study is that people are bad at predicting and observing their own time spent.
yes, I keep prompt plan logs
At the same time... that's not why I'm comfortable writing this. It's pretty obvious when you know what good vs bad feels like here and adjust accordingly:
1. Good: You are able to generate a long plan and that plan mostly works. These are big wins _as long as you are multitasking_: you are high throughput, even if the AI is slow. Think running 5-20min at a time for pretty good progress, for just a few minutes of your planning that you'd largely have to do anyways.
2. Bad: You are wasting a lot of attention chatting (so 1-2min runs) and repairing (re-planning from the top, vs progressing). There is no multitasking win.
It's pretty clear what situation you're in, with run duration on its own being a ~10X level difference.
Ex: I'll have ~3 projects going at the same time, and/or whatever else I'm doing. I'm not interacting "much" so I know it's a win. If a project is requiring interaction, well, now I need to jump in, and it's no longer agentic coding IMO, but chat assistant stuff.
At the same time, I power through case #2 in practice because we're investing in AI automation. We're retooling everything to enable long runs, so we'll still do the "hard" tasks via AI to identify & smooth the bumps. Similar to infrastructure-as-code and SDLC tooling, we're investing in automating as much of our stack as we can, so that means we figure out prompt templates, CI tooling, etc to enable the AI to do these so we can benefit later.
Re: Measuring the impact of AI on experienced open-source developer productivity
#359Here's the full paper, which has a lot of details missing from the summary linked above: https://metr.org/Early_2025_AI_Experienced_OS_Devs_Study.pdf My personal theory is that getting a significant productivity boost from LLM assistance and AI tools has a much steeper learning curve than most people expect. This study had 16 participants, with a mix of previous exposure to AI tools - 56% of them had never used Curso…
My personal experience was that of a decrease in productivity until I spent significant time with it. Managing configurations, prompting it the right way, asking other models for code reviews…
And I still see there is more I can unlock with more time learning the right interaction patterns.
For nasty, legacy codebases there is only so much you can do IMO. With green field (in certain domains), I become more confident every day that coding will be reduced to an AI task. I’m learning how to be a product manager / ideas guy in response
Re: Measuring the impact of AI on experienced open-source developer productivity
#360Hey HN, study author here. I'm a long-time HN user -- and I'll be in the comments today to answer questions/comments when possible! If you're short on time, I'd recommend just reading the linked blogpost or the announcement thread here [1], rather than the full paper. [1] https://x.com/METR_Evals/status/1943360399220388093
Does this reproduce for early/mid-career engineers who aren't at the top of their game?