It used to be that all you required to program was a computer and to RTFM but now we need to pay for API "tokens" and pray that there are no rug pull in the future.
Measuring the impact of AI on experienced open-source developer productivity
271–280 of 501 posts
Re: Measuring the impact of AI on experienced open-source developer productivity
#272Here's the full paper, which has a lot of details missing from the summary linked above: https://metr.org/Early_2025_AI_Experienced_OS_Devs_Study.pdf My personal theory is that getting a significant productivity boost from LLM assistance and AI tools has a much steeper learning curve than most people expect. This study had 16 participants, with a mix of previous exposure to AI tools - 56% of them had never used Curso…
> My personal theory is that getting a significant productivity boost from LLM assistance and AI tools has a much steeper learning curve than most people expect. This is what I heard about strong type systems (especially Haskell's) about 20-15 years ago. "History does not repeat, but it rhymes." If we rhyme "strong types will change the world" with "agentic LLMs will change the world," what do we get? My personal the…
I don't think that's a fair comparison. Type systems don't produce probabilistic output. Their entire purpose is to reduce the scope of possible errors you can write. They kind of did change the world, didn't they? I mean, not everyone is writing Haskell but Rust exists and it's doing pretty well. There was also not really a case to be made where type systems made software in general _worse_. But you could definitely make the case that LLM's might make software worse.
Re: Measuring the impact of AI on experienced open-source developer productivity
#273Here's the full paper, which has a lot of details missing from the summary linked above: https://metr.org/Early_2025_AI_Experienced_OS_Devs_Study.pdf My personal theory is that getting a significant productivity boost from LLM assistance and AI tools has a much steeper learning curve than most people expect. This study had 16 participants, with a mix of previous exposure to AI tools - 56% of them had never used Curso…
Any "tricks" you learn for one model may not be applicable to another, it isn't a given that previous experience with a company's product will increase the likelihood of productivity increases. When models change out from under you, the heuristics you've built up might be useless.
Re: Measuring the impact of AI on experienced open-source developer productivity
#274Earlier quoted context omitted.
> You’re not talking to the same people, you are talking to new people each time who now have had their threshold crossed. no, it's the same names, again and again
Got receipts? That sounds like a claim you could back up with a little bit of time spent using Hacker News search or similar. (I might try to get a tool like o3 to run those searches for me.)
try asking it what sealioning is
Re: Measuring the impact of AI on experienced open-source developer productivity
#275I actually think that pasting questions into chatGPT etc. and then getting general answers to put into your code is the way. “One shotting” apps, or even cursor and so forth seem like a waste of time. It feels like if you prompt it just right it might help but then it never really does.
I've done okay with copilot as a very smart autocomplete on: a) very typical codebase, with b) lots of boilerplate, where c) I'm not terribly familiar with the languages and frameworks, which are d) very, very popular but e) I don't really like, so I'm not particularly motivated to become familiar with them. I'm not a frontend developer, I don't like it, but I'm in a position now where I need to do frontend things with a verbose Typescript/React application which is not interesting from a technical point of view (good product, it's just not good because it has an interesting or demanding front end). Copilot (I use Emacs, so cursor is a non-starter, but copilot-mode works very well for Typescript) has been pretty invaluable to just sort of slogging through stuff.
For everything else, I think you're right, and actually the dialog-oriented method is way better. If I learn an approach and apply some general example from ChatGPT, but I do the typing and implementation myself so I need to understand what I'm doing, I'm actually leveling up and I know what I'm finished with. If I weren't "experienced", I'd worry about what it was doing to my critical thinking skills, but I know enough about learning on my own at this point to know I'm doing something.
I'm not interested in vibe coding at all--it seems like a one-way process to automate what was already not the hard part of software engineering; generating tutorial-level initial implementations. Just more scaffolding that eventually needs to be cleared away.
Re: Measuring the impact of AI on experienced open-source developer productivity
#276Earlier quoted context omitted.
Do you really see a massive jump? For context, I've been using AI, a mix of OpenAi + Claude, mainly for bashing out quick React stuff. For over a year now. Anything else it's generally rubbish and slower than working without. Though I still use it to rubber duck, so I'm still seeing the level of quality for backend. I'd say they're only marginally better today than they were even 2 years ago. Every time a new model c…
Yes. In January I would have told you AI tools are bullshit. Today I’m on the $200/month Claude Max plan. As with anything, your miles may vary: I’m not here to tell anyone that thinks they still suck that their experience is invalid, but to me it’s been a pretty big swing.
Ok, I'll have to try it out then. I've got a side project I've 3/4 finished and will let it loose on it.
So are you using Claude Code via the max plan, Cursor, or what?
I think I'd definitely hit AI news exhaustion and was viewing people raving about this agentic stuff as yet more AI fanbois. I'd just continued using the AI separate as setting up a new IDE seemed like too much work for the fractional gains I'd been seeing.
Re: Measuring the impact of AI on experienced open-source developer productivity
#277So they paid developers 300 x 246 = about 73K just for developer recruitment for the study, which is not in any academic journal, or has no peer reviews? The underlying paper looks quite polished and not overtly AI generated so I don't want to say it entirely made up, but how were they even able to get funding for this?
>which is not in any academic journal, or has no peer reviews? As a philosopher who is into epistemology and ontology, I find this to be as abhorrent as religion. 'Science' doesn't matter who publishes it. Science needs to be replicated. The psychology replication crisis is a prime example of why peer reviews and publishing in a journal matters 0.
Peer reviews are very important to filter out obviously low effort stuff.
...Or should I say "were" very important? With the help of today's GenAI every low effort stuff can look high effort without much extra effort.
Re: Measuring the impact of AI on experienced open-source developer productivity
#278Earlier quoted context omitted.
Invoking personality is to the behavioral science as invoking God is to the natural sciences. One can explain anything by appealing to personality, and as such it explains nothing. Psychologists have been trying to make sense of personality for over a century without much success (the best efforts so far have been a five factor model [Big 5] which has ultimately pretty minor predictive value), which is why most behav…
Fair comment, but I'm not down with behavioralism, and people have personalities, regrettably.
This is still ultimately a research within the field of the behavior sciences, and as such the laws of human behavior apply, where behaviorism offers a far more successful theoretical framework than personality psychology.
Nobody is denying that people have personalities btw. Not even true behavioralists do that, they simply argue from reductionism that personality can be explained with learning contingencies and the reinforcement history. Very few people are true behavioralists these days though, but within the behavior sciences, scientists are much more likely to borrow missing factors (i.e. things that learning contingencies fail to explain) from fields such as cognitive science (or even further to neuroscience) and (less often) social science.
What I am arguing here, however, is that the appeal to personality is unnecessary when explaining behavior.
As for figuring out what personality is, that is still within the realm of philosophy. Maybe cognitive science will do a better job at explaining it than psychometricians have done for the past century. I certainly hope so, it would be nice to have a better model of human behavior. But I think even if we could explain personality, it still wouldn’t help us here. At best we would be in a similar situation as physics, where one model can explain things traveling at the speed of light, while another model can explain things at the sub-atomic scale, but the two models cannot be applied together.
Re: Measuring the impact of AI on experienced open-source developer productivity
#279My hot take: Cursor is a bad tool for agentic coding. Had a subscription and canceled it in favor of Claude Code. I don’t want to spend 90% of my time approving every line the agent wants to write. With Claude Code I review whole diffs - 1-2 minutes of the agent’s work at a time. Then I work with the agent at a level of what its approach is, almost never asking about specific lines of code. I can look at 5 files at once in git diff and then ask “why’d you choose that way?” “Can we undo that and try to find a simpler way?”
Cursor’s workflow exposes how differently different people track context. The best ways to work with Cursor may simply not work for some of us.
If Cursor isn’t working for you, I strongly encourage you to try CLI agents like Claude Code.
Re: Measuring the impact of AI on experienced open-source developer productivity
#280Earlier quoted context omitted.
Yes. In January I would have told you AI tools are bullshit. Today I’m on the $200/month Claude Max plan. As with anything, your miles may vary: I’m not here to tell anyone that thinks they still suck that their experience is invalid, but to me it’s been a pretty big swing.
Ok, I'll have to try it out then. I've got a side project I've 3/4 finished and will let it loose on it. So are you using Claude Code via the max plan, Cursor, or what? I think I'd definitely hit AI news exhaustion and was viewing people raving about this agentic stuff as yet more AI fanbois. I'd just continued using the AI separate as setting up a new IDE seemed like too much work for the fractional gains I'd been s…
I had a bad time with Cursor. I use Claude Code inside of VS: Code. You don't necessarily need Max, but you can spend a lot of money very quickly on API tokens, so I'd recommend to anyone trying, start with the $20/month one, no need to spend a ton of money just to try something out.
There is a skill gap, like, I think of it like vim: at first it slows you down, but then as you learn it, you end up speeding up. So you may also find that it doesn't really vibe with the way you work, even if I am having a good time with it. I know people who are great engineers who still don't like this stuff, just like I know ones that do too.