Earlier quoted context omitted.
Got receipts? That sounds like a claim you could back up with a little bit of time spent using Hacker News search or similar. (I might try to get a tool like o3 to run those searches for me.)
try asking it what sealioning is
Measuring the impact of AI on experienced open-source developer productivity
331–340 of 501 posts
Re: Measuring the impact of AI on experienced open-source developer productivity
#332Here's the full paper, which has a lot of details missing from the summary linked above: https://metr.org/Early_2025_AI_Experienced_OS_Devs_Study.pdf My personal theory is that getting a significant productivity boost from LLM assistance and AI tools has a much steeper learning curve than most people expect. This study had 16 participants, with a mix of previous exposure to AI tools - 56% of them had never used Curso…
I find the very popular response of "you're just not using it right" to be big copout for LLMs, especially at the scale we see today. It's hard to think of any other major tech product where it's acceptable to shift so much blame on the user. Typically if a user doesn't find value in the product, we agree that the product is poorly designed/implemented, not that the user is bad. But AI seems somehow exempt from this…
If my phone keeps crashing or if the browser is slow or clunky then yes, it’s not on me, it’s the phone, but an LLM is a lot more open ended in what it can do. Unlike the phone example above where I expect it to work from a simple input (turning it on) or action (open browser, punch in a url), what an LLM does is more complex and nuanced.
Even the same prompt from different users might result in different output - so there is more onus on the user to craft the right input.
Perhaps that’s why AI is exempt for now.
Re: Measuring the impact of AI on experienced open-source developer productivity
#333Earlier quoted context omitted.
Fair enough. For what it's worth, I've always thought that the more reasonable claim is that AI tools make poor-average developers more productive, not necessarily expert developers.
Personally I don't want poor-average developers to be more productive, I want them to be more expert
Re: Measuring the impact of AI on experienced open-source developer productivity
#334Earlier quoted context omitted.
Still odd to me that the only vibe coded software that gets aquired are by companies selling tools or want to promote vibe coding.
Pardon my caps, but WHO CARES about acquisitions?! You’ve been given a dubiously capable genie that can write code without you having to do it! If this thing can build first drafts of those side projects you always think about and never get around to, that in and of itself is useful! If it can do the yak-shaving required to set up those e2e tests you know you should have but never have time for it is useful! Have it…
Took me a week to build those tools. Its much more reliable (and flexible) than any LLM and cost me nothing.
It comes with secure Auth, email, admin, ect ect.. Doesn't cost me a dime and almost never has a common vulnerability.
Best part about it. I know how my side project runs.
Re: Measuring the impact of AI on experienced open-source developer productivity
#335Here's the full paper, which has a lot of details missing from the summary linked above: https://metr.org/Early_2025_AI_Experienced_OS_Devs_Study.pdf My personal theory is that getting a significant productivity boost from LLM assistance and AI tools has a much steeper learning curve than most people expect. This study had 16 participants, with a mix of previous exposure to AI tools - 56% of them had never used Curso…
I find the very popular response of "you're just not using it right" to be big copout for LLMs, especially at the scale we see today. It's hard to think of any other major tech product where it's acceptable to shift so much blame on the user. Typically if a user doesn't find value in the product, we agree that the product is poorly designed/implemented, not that the user is bad. But AI seems somehow exempt from this…
All take quite an effort to master, until then they might slow one down or outright kill.
Re: Measuring the impact of AI on experienced open-source developer productivity
#336We probably need a bit less wishful thinking. Blindly trusting what the AI suggests tends to backfire. The real challenge is figuring out where it actually helps, and where it quietly gets in the way.
Re: Measuring the impact of AI on experienced open-source developer productivity
#337Re: Measuring the impact of AI on experienced open-source developer productivity
#338Here's the full paper, which has a lot of details missing from the summary linked above: https://metr.org/Early_2025_AI_Experienced_OS_Devs_Study.pdf My personal theory is that getting a significant productivity boost from LLM assistance and AI tools has a much steeper learning curve than most people expect. This study had 16 participants, with a mix of previous exposure to AI tools - 56% of them had never used Curso…
I find the very popular response of "you're just not using it right" to be big copout for LLMs, especially at the scale we see today. It's hard to think of any other major tech product where it's acceptable to shift so much blame on the user. Typically if a user doesn't find value in the product, we agree that the product is poorly designed/implemented, not that the user is bad. But AI seems somehow exempt from this…
Re: Measuring the impact of AI on experienced open-source developer productivity
#339One thing I've experienced in trying to use LLMs to code in an existing large code base is that it's _extremely_ hard to accurately describe what you want to do. Oftentimes, you are working on a problem with a web of interactions all over the code and describing the problem to an LLM will take far longer than just doing it manually. This is not the case with generating new (boilerplate) code for projects, which is wh…
Re: Measuring the impact of AI on experienced open-source developer productivity
#340For me, the measurable gain in productiviy comes when I am working with a new language or new technology. If I were to use claude code to help implement a feature of a python library I've worked on for years then I don't think it would help much (Maybe even hurt). However, if I use claude code on some go code I have very little experience with, or using it to write/modify helm charts then I can definitely say it spee…
That said, this comes up often in my office. It's just not giving really good advice in many situations - especially novel ones.
AI is super good at coming up with things that have been written ad nauseam for coding-interview-prep website