Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Did Claude increase bugs in rsync?

alexispurslane.github.io

Post reply on HN
Print view View original

381–390 of 611 posts

Re: Did Claude increase bugs in rsync?

#381

Post by tolciho »

jonquark wrote: post #117
logicprog wrote: post #103

Earlier quoted context omitted.

Your first and second points seem to contradict each other because if all of the bugs for 3.4.1 should be attributed to 3.4.0, that pushes the timetable back even further that unattributed LLM commits would have to have been being committed to the project, which just makes your point even more absurd. Which brings me to my overall response, which is that there is absolutely no evidence, and nothing even intimating th…

Isn't the metric that you've used "bugs per commit ~ per new line of code" going to miss the issue? All code is technical debt. If rsync releases used to have 500 lines changed and 5 bugs in and AI-powered rsync releases have 50000 lines and 500 bugs, it's the same bugs/line but much worse experience for the user? I've not looked into the details of this case and I do use AI assistance coding at work but in my experi…

OpenBSD used to have sqlite in base, but the code churn rate was too high to review. This was well before the recent LLM craze, so a human (perhaps not a normal one, though) already sufficies to generate too many changes for others to check for errors.

Re: Did Claude increase bugs in rsync?

#382

Post by MYEUHD »

logicprog wrote: post #178
MYEUHD wrote: post #170

Earlier quoted context omitted.

There is openrsync, which the OpenBSD re-implementation of rsync It's not a fork, but it's 8 years old, and is already shipped by default in OpenBSD and macOS.

To quote Tridge: > As to all the people saying “I’m going to package openrsync for platform XXX and we’ll use that!”. I find that rather amusing. If you do decide to go down that path I’d suggest you try the new rsync test suite on openrsync if you can stomach something that an AI has helped write. I tried it today and openrsync currently fails 85 of 98 tests, so I’m sure it won’t take you long to get it up to speed.…

I have already been using openrsync even before the recent AI drama.

Just like I have been using doas for several years.

All I need is `rsync -urvP` and I suspect the majority of users don't need the advanced features either.

The smaller code base also means less bugs and vulnerabilities. As an example doas is ~1k lines vs 160k for sudo. That surely means a smaller attack surface. The same is true for openrsync and rsync at approximately 18k vs 57k lines.

Re: Did Claude increase bugs in rsync?

#383

Post by kelipso »

Eufrat wrote: post #370
rsc wrote: post #354

Earlier quoted context omitted.

As a long-time open-source maintainer, I find all the second-guessing and armchair psychoanalysis here (not just in this comment, all over HN) about Tridge's motivations, state of mind, and so on incredibly off-putting. Tridge doesn't owe anyone anything as far as rsync is concerned. Yet he is spending his time maintaining it, only to be attacked for his efforts. To respond to the specific technical point, there real…

> As a long-time open-source maintainer, I find all the second-guessing and armchair psychoanalysis here (not just in this comment, all over HN) about Tridge's motivations, state of mind, and so on incredibly off-putting. I agree that the entire episode is obscene, but I am also unsure of what to do here either. On some level this is the same problem movie stars run into. I agree that guessing or waxing about the mot…

Yeah the maintainers don’t owe users nothing is a disgusting sentiment that doesn’t stand real scrutiny. There is a social contract here. If you want to be respected and get recognized as “tridge” or whatever your name is, you owe the people that recognize you and that wider community in general.

Re: Did Claude increase bugs in rsync?

#384

Post by red75prime »

eschaton wrote: post #372
red75prime wrote: post #371

Earlier quoted context omitted.

> I found exact matches. Were those functions trivial? With, say, 1% probability of someone who have not seen them writing them like that? > Just like LLMs have reproduced chapters of books and articles from the New York Times exactly. Have you read the articles? As far as I remember they fed large chunks of an article multiple times to an LLM to sometimes get a not-so-long exact match. It can mean that LLMs can infe…

No, the functions weren’t trivial, and a lot of the surrounding code and structure bore substantial similarities as well. If you saw the two files next to each other, you’d assume it was the result of a copy-paste-adjust process if you didn’t know an LLM was involved.

I can only speculate that the model that generated the code hasn't undergone selective unlearning for verbatim data (SUV) or something similar. As you understand "sometimes generates verbatim code" and "just regurgitates [non-trivial] portions its input" are different statements.

The possibility of SUV clearly shows that a model does more than "just regurgitating."

Re: Did Claude increase bugs in rsync?

#385

Post by dnnddidiej »

matheusmoreira wrote: post #285
RustyRussell wrote: post #258

For those commenting, I suggest you read the post linked by the rsync author: https://medium.com/@tridge60/rsync-and-outrage-d9849599e5a0 (Disclosure: while I haven't talked with him in years, Tridge was my colleague and mentor for many years. I feel it is worth considering his view before joining a crusade)

This should be the top comment. I think it's pretty sad that he even had to write it. Quite a lot of judgement from people who aren't paying his bills.

The title at least sounds less like judgement and more analysis and more about AI assistance (and claude in particular) than rsync. Maybe I am too used to postmortems!

Re: Did Claude increase bugs in rsync?

#386

Post by matheusmoreira »

Barrin92 wrote: post #380
matheusmoreira wrote: post #373

Earlier quoted context omitted.

I don't know, am I? Why don't you check out my work and decide for yourself? Better than forming prejudiced opinions about others.

>Why don't you check out my work and decide for yourself? because no person can read every line of code written in software they use, or track every commit made to a project. Integrity and authorship matters. If a person lies or obfuscates the origin of what they produce, an article, software, what have you they're doing it for a reason, otherwise they would be honest. That's not prejudice, that's recognizing deceit.…

> because no person can read every line of code written in software they use, or track every commit made to a project

Ask Claude to do it for you.

> they're doing it for a reason

And you concluded that the reason was they were pretenders who can't hack it.

That's your prejudice. Not interested in helping you categorize me, thanks.

Re: Did Claude increase bugs in rsync?

#387

Post by tptacek »

kelipso wrote: post #383
Eufrat wrote: post #370

Earlier quoted context omitted.

> As a long-time open-source maintainer, I find all the second-guessing and armchair psychoanalysis here (not just in this comment, all over HN) about Tridge's motivations, state of mind, and so on incredibly off-putting. I agree that the entire episode is obscene, but I am also unsure of what to do here either. On some level this is the same problem movie stars run into. I agree that guessing or waxing about the mot…

Yeah the maintainers don’t owe users nothing is a disgusting sentiment that doesn’t stand real scrutiny. There is a social contract here. If you want to be respected and get recognized as “tridge” or whatever your name is, you owe the people that recognize you and that wider community in general.

No there isn't.

Re: Did Claude increase bugs in rsync?

#388

Post by cobertos »

logicprog wrote: post #280
cobertos wrote: post #265

This post just gives me more questions than answers and I'm unable to form a decision: * Why was v3.4.1 the most buggy, right before the Claude commits? Why did "nobody notice"? It's way to strange to just say welp, it must be human error. * Why does v3.4.2 have 0 bugs, or 0 bug score. And why was such an outlier (no other commit seemingly has this??) allowed to mix into aggregate statistics and bring all the "is Cla…

> Why was v3.4.1 the most buggy, right before the Claude commits? Why did "nobody notice"? It's way to strange to just say welp, it must be human error. Why wouldn't it be except question begging priors assuming it couldn't be? > Why does v3.4.2 have 0 bugs, or 0 bug score. And why was such an outlier (no other commit seemingly has this??) allowed to mix into aggregate statistics and bring all the "is Claude buggy?"…

> Why wouldn't it be except question begging priors assuming it couldn't be?

A multitude of reasons? A change in maintainer. A change in the mental state of a maintainer. A sudden focus by the community on a given undesirable behavior. Someone else here suggested use of Claude AI before it was disclosured. The framing implies that it was human-produced coding error, but my point is it could be _any other human error_ or even just some odd benign human behavior (a stampede of bug submitters), affecting the data. Which does not lead to the conclusion that AI code > human code. Not looking at these potentials is so unsatisfying.

> My original metrics which didn't filter out feature requests...

It still feels like a lot of weight of the phrase "If that doesn't look like a red flag to you, you'd be right." hinges on the fact that one of the versions has 0 bugs and it really killed the weight of that statement for me, because the oddity of there being 0 bugs just wasn't explained.

---

Could you please post the duckdb file that has the raw bug -> severity + version mapping to the GitHub repo? I have a desire to dig into this myself

Re: Did Claude increase bugs in rsync?

#389

Post by e40 »

While I'm grateful for all Andrew has done to create and maintain rsync, I rely heavily on it for backing up files between machines on my home network, so I've spent the time to figure out how to pin the Homebrew version of rsync to 3.4.1 because the bugs in the subsequent two versions really scare me (as does the original report that triggered all this).

Here is the process I used to do it, which was way more complex than I thought it would be:

https://gist.github.com/e40/caa67c1b8d439a528695f996d0519d8e

Re: Did Claude increase bugs in rsync?

#390

Post by ex-aws-dude »

scsh wrote: post #36

> It does not control for commit complexity, security intensity, or bug severity. It does not distinguish between a one-line typo fix and a CVE patch. It is a blunt instrument. But the critics' accusation is also blunt: "Claude is making things worse." A blunt instrument is the fairest response. If by fairest you mean to say that this analysis and response is sufficient, then I'm sorry but I have to disagree. We real…

The burden of proof is on the one making the claim?
Post reply on HN