Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Did Claude increase bugs in rsync?

alexispurslane.github.io

Post reply on HN
Print view View original

411–420 of 611 posts

Re: Did Claude increase bugs in rsync?

#411

Post by its-summertime »

PunchyHamster wrote: post #165
aesthesia wrote: post #95

I don't have a dog in this fight, but a few points that look a little suspicious: - The release with the highest number of attributed bugs is the release _right before_ the first release with Claude-coauthored commits, released in January; is there a chance that unattributed LLM-authored commits made it into this release? - The release attribution methodology is not great, since it will tend to attribute bugs introdu…

Let's start with most outright alarming error - the claude statistics are taken out of whole 2 data points

If one asks "Is the house on 123 Road Street, NJ, taller than the statistical average", then that there is only 1 datapoint for the house on 123 Road Street, NJ. Which is also 100% of the houses on 123 Road Street, NJ.

Re: Did Claude increase bugs in rsync?

#412

Post by lelanthran »

nomel wrote: post #264
jarym wrote: post #256

I've been coding for over 2 decades. I love it, I've always loved it and I likely always will. I was an AI skeptic some months ago but truly Claude and Codex have changed my development style and velocity in a way I never imagined would ever be possible. With that, yes, I produce more code and am finding more bugs. So looking over at comments in HN articles the amount of polarising hate to anything produced with AI i…

I've always noticed, within any subject involving tools, there are people who like the tools, and some people who like to use the tools to do something else. With programming, I've always been in the later: it's a tool that allows me to do what I actually love, which is problem solving, system level thinking , and providing some nice solution to that problem, that happens to be through software. So, I have an absolut…

> So, I have an absolute blast with AI, because it helps do the more boring bits.

So... you're vibing? Not looking at the code at all?

Re: Did Claude increase bugs in rsync?

#413

Post by JetSetIlly »

rsc wrote: post #354
Eufrat wrote: post #327

Earlier quoted context omitted.

I think Tridge is simultaneously trying to be proactive and kinda giving too much credit to marketing. Anthropic has not been able to really give numbers or actual values on what Mythos can really do. It just waved Mythos in front of the public like a boogeyman screaming that AI is going to cause a security nightmare (and it has, but mostly through vibe coded trash from what I’ve noticed); I’m hard pressed to find th…

As a long-time open-source maintainer, I find all the second-guessing and armchair psychoanalysis here (not just in this comment, all over HN) about Tridge's motivations, state of mind, and so on incredibly off-putting. Tridge doesn't owe anyone anything as far as rsync is concerned. Yet he is spending his time maintaining it, only to be attacked for his efforts. To respond to the specific technical point, there real…

> As a long-time open-source maintainer, I find all the second-guessing and armchair psychoanalysis here (not just in this comment, all over HN) about Tridge's motivations, state of mind, and so on incredibly off-putting.

Much of the language from both groups is incredibly off-putting, frankly. Tridge in his blog post describes people as "foaming at the mouth"?!

The rhetoric around this has gotten way too emotional from both groups.

I'm glad I'm just a hobbyist.

Re: Did Claude increase bugs in rsync?

#414

Post by jrflowers »

logicprog wrote: post #337
jrflowers wrote: post #306

Earlier quoted context omitted.

The only reason why people are talking about this is because of the bugs in the code that the chat bot generated OP. People updated to a version of rsync that didn’t work right, the one with all the bot commits in it. This blog post is about how claude didn’t create more bugs than usual if you think about it in one very specific way, not that it didn’t create more bugs at all. It is like if your neighbor opens your d…

That doesn't make any sense, what?

I am not sure what to tell you here, because somebody literally posted the code of a bug that Claude inserted here

https://news.ycombinator.com/item?id=48419197

And your response to someone pointing out that sloppy, buggy code that Claude introduced, was to just quote Tridge (which does not in any way refute the fact that you’re looking at a bug that Claude introduced to the code)

https://news.ycombinator.com/item?id=48419621

I’m not entirely sure what the purpose of this project is (maybe to “prove” Tridge’s opinions about LLMs and human intelligence that he made in the linked blog post to be right?), but it appears as though you are ignoring irrefutably true observations. You just asserted that “the data” doesn’t show Claude introducing any bugs (which is a bizarre claim) after previously responding to a documented bug with a… deferral? Do bugs not count if you can find a vague excuse for it?

There is nothing in the blog post that is evidence that Claude didn’t introduce bugs. It is a thought experiment that uses “increase bugs” and “increase bugs more than a given arbitrary statical amount that I selected” as interchangeable statements.

Re: Did Claude increase bugs in rsync?

#415

Post by bonzini »

rakel_rakel wrote: post #398
rsc wrote: post #354

Earlier quoted context omitted.

As a long-time open-source maintainer, I find all the second-guessing and armchair psychoanalysis here (not just in this comment, all over HN) about Tridge's motivations, state of mind, and so on incredibly off-putting. Tridge doesn't owe anyone anything as far as rsync is concerned. Yet he is spending his time maintaining it, only to be attacked for his efforts. To respond to the specific technical point, there real…

I agree, it's very off-putting, and I totally understand that the amount of reports are overwhelming for maintainers of popular libraries. > More reports means more fixes means more code changes means more bugs. Sounds like we'll be riding a downward spiral for the foreseeable future? It will be very interesting to see how stats like the ones you shared develop in the coming year(s). From the article I find this a bi…

What Tridge says is that the "more code" is more fixes and more thorough test suites, not random changes made by LLMs.

Re: Did Claude increase bugs in rsync?

#416

Post by [deleted] »

faitswulff wrote: post #22

> The analysis uses a single metric: bugs per 10 commits (bugs/10c). Bugs per commit as a metric papers over severity, both in terms of security severity as well as the effect on the user. A mislabeled button has the same weight as the entire app crashing in this framework.

[deleted]

Re: Did Claude increase bugs in rsync?

#417

Post by nasretdinov »

Regardless of the claims made in this analysis, I've personally observed that there are indeed more bugs (or more subtle issues, like nonsensical error messages) being shipped when using LLMs, but not _really_ because LLMs suck, but because you're spending less time thinking about the problem, and you subsequently miss more edge cases, etc.

The best approach I've tried that actually increases quality (and _may_ speed up development) is to write ~80% of the code yourself and then ask LLM to review it thoroughly. While it's doing its thing you're also thinking about the code and reviewing it yourself in parallel. You then merge the findings and fix stuff worth fixing. At this point the authorship of the code is still mostly yours, you _understand_ the system and you ship fewer bugs, slightly faster than otherwise. It's a moderate improvement to the workflow, but it actually doesn't cost nearly as much either, and definitely doesn't produce rage at the machine from the slop. The only downside is that it requires lots of discipline, and it's a relatively rare commodity among software engineers these days.

Re: Did Claude increase bugs in rsync?

#418

Post by bsza »

logicprog wrote: post #140
faitswulff wrote: post #22

> The analysis uses a single metric: bugs per 10 commits (bugs/10c). Bugs per commit as a metric papers over severity, both in terms of security severity as well as the effect on the user. A mislabeled button has the same weight as the entire app crashing in this framework.

I've now resolved this. The new version, which should be live on GH Pages soon, uses — what I think is — a pretty good methodology for assigning severity to each bug, normalizes it to 0.0-1.0, sums that, and treats that as the total severity weighted bugs, then does the analysis based on that. It did not change the analysis in any material way.

No Claude, it still makes zero sense as a metric.

A commit is a measure of nothing. Severity weighted bugs per unit of nothing? What does that even mean? In any repo it's trivial to achieve a sev/10c that's arbitrarily close to zero while completely ruining everything.

I suggest you practice some humility and update your conclusion instead of updating the mental gymnastics you used to arrive at the same conclusion.

Re: Did Claude increase bugs in rsync?

#419

Post by Eufrat »

JetSetIlly wrote: post #413
rsc wrote: post #354

Earlier quoted context omitted.

As a long-time open-source maintainer, I find all the second-guessing and armchair psychoanalysis here (not just in this comment, all over HN) about Tridge's motivations, state of mind, and so on incredibly off-putting. Tridge doesn't owe anyone anything as far as rsync is concerned. Yet he is spending his time maintaining it, only to be attacked for his efforts. To respond to the specific technical point, there real…

> As a long-time open-source maintainer, I find all the second-guessing and armchair psychoanalysis here (not just in this comment, all over HN) about Tridge's motivations, state of mind, and so on incredibly off-putting. Much of the language from both groups is incredibly off-putting, frankly. Tridge in his blog post describes people as "foaming at the mouth"?! The rhetoric around this has gotten way too emotional f…

IMHO, the whole episode is just embarrassing. I have no doubt he’s just trying to do the right thing. You can disagree with the tactics, but the vitriol is outrageous. rsync is a gift to the world and we should be grateful and mindful of how much it has been quietly woven into the fabric of computing. rsync is taken for granted. This is not okay.

Re: Did Claude increase bugs in rsync?

#420

Post by wolletd »

Eufrat wrote: post #327
logicprog wrote: post #257

Earlier quoted context omitted.

Tridge already explains this: "Like many developers of open source packages I’ve been hit by a flood of security reports lately in my role as the rsync maintainer. Many of those reports are AI generated (not all though, there are some notable ones with very careful and high quality manual analysis). As this flood started to get more intense I realised I needed to raise the defences on rsync a lot — we needed much mor…

I think Tridge is simultaneously trying to be proactive and kinda giving too much credit to marketing. Anthropic has not been able to really give numbers or actual values on what Mythos can really do. It just waved Mythos in front of the public like a boogeyman screaming that AI is going to cause a security nightmare (and it has, but mostly through vibe coded trash from what I’ve noticed); I’m hard pressed to find th…

> “I’d rather be out sailing than working on rsync security issues, so I have reached for several AI tools to help with what needs to be done,”

Well, then maybe it's already overdue to find a new maintainer for the project and let someone else continue it? The tool will not get better from someone working on it who doesn't want to.

Post reply on HN