Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

AI slows down open source developers. Peter Naur can teach us why

johnwhiles.com

Post reply on HN
Print view View original

41–50 of 217 posts

Re: AI slows down open source developers. Peter Naur can teach us why

#41

Post by joshmarlow »

I've gotten some pretty cool things working with LLMs doing most of the heavy lifting using the following approaches:

* spec out project goals and relevant context in a README and spec out all components; have the AI build out each component and compose them. I understand the high-level but don't necessarily know all of the low-level details. This is particularly helpful when I'm not deeply familiar with some of the underlying technologies/libraries. * having an AI write tests for code that I've verified is working. As we all know, testing is tedious - so of course I want to automate it. And we written tests (for well written code) can be pretty easy to review.

Re: AI slows down open source developers. Peter Naur can teach us why

#42

Post by steveklabnik »

rosspackard wrote: post #10

One mediocre paper/study (it should not even be called that with all the bias and sample size issues) and now we have to put up with stories re-hashing and dissecting it. I really hope these don't get upvoted more in the future. 16 devs. And they weren't allowed to pick which tasks they used the AI on. Ridiculous. Also using it on "old and >1 million line" codebases and then extrapolating that to software engineering…

> and then extrapolating that to software engineering in general.

To the credit of the paper authors, they were very clear that they were not making a claim against software engineering in general. But everyone wants to reinforce their biases, so...

Re: AI slows down open source developers. Peter Naur can teach us why

#43

Post by crinkly »

piker wrote: post #37

My main two attempts at using an “agentic” coding workflow were trying to incorporate an Outlook COM interface into my rust code base and to streamline an existing abstract windows API interaction to avoid copying memory a couple of times. Both wasted tremendous amounts of time and were ultimately abandoned leaving me only slightly more educated about windows development. They make great autocompletion engines but I…

This is typically what I see when I’ve seen it applied. And as always trying to hammer nails in with a banana.

Rather than fit two generally disparate things together it’s probably better to just use VSTO and C# (hammer and nails) rather than some unholy combination no one else has tried or suffered through. When it goes wrong there’s more info to get you unstuck.

Re: AI slows down open source developers. Peter Naur can teach us why

#44

Post by rosspackard »

jplusequalt wrote: post #30
rosspackard wrote: post #10

One mediocre paper/study (it should not even be called that with all the bias and sample size issues) and now we have to put up with stories re-hashing and dissecting it. I really hope these don't get upvoted more in the future. 16 devs. And they weren't allowed to pick which tasks they used the AI on. Ridiculous. Also using it on "old and >1 million line" codebases and then extrapolating that to software engineering…

>One mediocre paper/study (it should not even be called that with all the bias and sample size issues) Can you bring up any specific issues with the metr study? Alternatively, can you site a journal that critiques it?

It was just published. Too new for someone to conduct a direct study to critique and journals don't just publish critiques anyway. It would have to be a study that disputes the results.

They used 16 developers. The confidence intervals are wide and a few atypical issues per dev could swing the headline figure.

Veteran maintainers on projects they know inside-out. This is a bias.

Devs supplied the issue list (then randomized) which still leads to subtle self-selection bias. Maintainers may pick tasks they enjoy or that showcase deep repo knowledge—exactly where AI probably has least marginal value.

Time was not independently logged and was self-reported.

No possible direct quality metric is possible. Could the AI code be better?

The Hawthorne effect. Knowing they are observed paid may make devs over-document, over-prompt, or simply take their time.

Many of the devs were new to Cursor

Bias in forecasting.

Re: AI slows down open source developers. Peter Naur can teach us why

#45

Post by Vegenoid »

nico wrote: post #8

> They are experienced open source developers, working on their own projects I just started working on a 3-month old codebase written by someone else, in a framework and architecture I had never used before Within a couple hours, with the help of Claude Code, I had already created a really nice system to replicate data from staging to local development. Something I had built before in other projects, and I new that m…

I have had similar experiences as you, but this is not the kind of work that the study is talking about:

“When open source developers working in codebases that they are deeply familiar with use AI tools to complete a task, they take longer to complete that task”

I have anecdotally found this to be true as well, that an LLM greatly accelerates my ramp up time in a new codebase, but then actually leads me astray once I am familiar with the project.

Re: AI slows down open source developers. Peter Naur can teach us why

#46

Post by doc_manhat »

horsawlarway wrote: post #15
doc_manhat wrote: post #3

I directionally disagree with this: ``` It's common for engineers to end up working on projects which they don't have an accurate mental model of. Projects built by people who have long since left the company for pastures new. It's equally common for developers to work in environments where little value is placed on understanding systems, but a lot of value is placed on quickly delivering changes that mostly work. In…

Just anecdotally - I think your reason for disagreeing is a valid statement, but not a valid counterpoint to the argument being made. So > Reason: you cannot evaluate the work accurately if you have no mental model. If there's a bug given the systems unwritten assumptions you may not catch it. This is completely correct. It's a very fair statement. The problem is that a developer coming into a large legacy project is…

Yeah fair points particularly for larger codebases I could see this being a huge time saver.

Re: AI slows down open source developers. Peter Naur can teach us why

#47

Post by rosspackard »

steveklabnik wrote: post #42
rosspackard wrote: post #10

One mediocre paper/study (it should not even be called that with all the bias and sample size issues) and now we have to put up with stories re-hashing and dissecting it. I really hope these don't get upvoted more in the future. 16 devs. And they weren't allowed to pick which tasks they used the AI on. Ridiculous. Also using it on "old and >1 million line" codebases and then extrapolating that to software engineering…

> and then extrapolating that to software engineering in general. To the credit of the paper authors, they were very clear that they were not making a claim against software engineering in general. But everyone wants to reinforce their biases, so...

Great for the authors. But everyone else seems to be extrapolating. Authors have a responsibility and should recognize how their work will be used.

Metr may overall have an ok mission, but their motivation is questionable. They published something like this to get attention. Mission accomplished on that but they had to have known how this would be twisted.

Re: AI slows down open source developers. Peter Naur can teach us why

#48

Post by [deleted] »

gjsman-1000 wrote: post #4

What I thought was fascinating, and should be a warning sign to everyone here: Before beginning the study, the average developer expected about a 20% productivity boost. After ending the study, the average developer (potentially: you) believed they actually were 20% more productive. In reality, they were 0% more productive at best, and 40% less productive at worst. Think about what it would be like to be that develop…

[deleted]

Re: AI slows down open source developers. Peter Naur can teach us why

#49

Post by rosspackard »

sumeno wrote: post #14
rosspackard wrote: post #10

One mediocre paper/study (it should not even be called that with all the bias and sample size issues) and now we have to put up with stories re-hashing and dissecting it. I really hope these don't get upvoted more in the future. 16 devs. And they weren't allowed to pick which tasks they used the AI on. Ridiculous. Also using it on "old and >1 million line" codebases and then extrapolating that to software engineering…

> And they weren't allowed to pick which tasks they used the AI on. They were allowed to pick whether or not to use AI on a subset of tasks. They weren't forced to use AI on tasks that don't make sense for AI

Half the tasks they were not allowed to use AI.

Re: AI slows down open source developers. Peter Naur can teach us why

#50

Post by antimora »

I'm one of the regular code reviewers for Burn (a deep learning framework in Rust). I recently had to close a PR because the submitter's bug fix was clearly written entirely by an AI agent. The "fix" simply muted an error instead of addressing the root cause. This is exactly what AI tends to do when it can't identify the actual problem. The code was unnecessarily verbose and even included tests for muting the error. Based on the person's profile, I suspect their motivation was just to get a commit on their record. This is becoming a troubling trend with AI tools.
Post reply on HN