Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Did Claude increase bugs in rsync?

alexispurslane.github.io

Post reply on HN
Print view View original

171–180 of 611 posts

Re: Did Claude increase bugs in rsync?

#171

Post by logicprog »

PunchyHamster wrote: post #156

The fact last few commits were attributed to claude doesn't mean previous ones didn't use it. Also if you write a paper where you get statistical conclusions out of whole 2 datapoints you'd be laughed out of the room

> Also if you write a paper where you get statistical conclusions out of whole 2 datapoints you'd be laughed out of the room

I'm using methods appropriate to that low amount of data, first of all. Second of all, since I'm only trying to show there's no evidence for the anti-AI hypothesis (not disprove it, or prove the null hypothesis), that's sufficient in itself. Also, I wonder why nobody said things like you're saying ("there's too little data to tell") in response to all the absolutist claims that AI caused rsync to get worse?

> The fact last few commits were attributed to claude doesn't mean previous ones didn't use it.

At this point, you're just positing Russel's Teapot: you'll keep assuming more and more of the code was "secretly" Claude when there's no evidence for it and no reason to think so, just because you've started with the assumption that Claude makes things worse and you want to find a way to prove it.

Re: Did Claude increase bugs in rsync?

#172

Post by themafia »

> If anyone complains about my verbosity or sentence structure — as they usually do, which is the reason I originally let the AI write the prose, among other reasons obsoleted by templating — they can go fuck themselves.

You can write for an audience or you can write for yourself. Which is fine either way but you shouldn't pass the blame for bad results on to your audience.

> and recieving almost no substantive input, discussion, or response on the actual content of the article

Well did you write it for that purpose?

> "Just wait, more bugs will surface" -- v3.4.3 has been out long enough

Wait for _more releases_. As your own data shows the bug rate is not consistent between releases. So this is probably not a worthwhile metric. Perhaps systems touched, new features included, or attempted fixes would be a better way to contextualize releases and the goals of the author.

Re: Did Claude increase bugs in rsync?

#173

Post by runarberg »

PunchyHamster wrote: post #165
aesthesia wrote: post #95

I don't have a dog in this fight, but a few points that look a little suspicious: - The release with the highest number of attributed bugs is the release _right before_ the first release with Claude-coauthored commits, released in January; is there a chance that unattributed LLM-authored commits made it into this release? - The release attribution methodology is not great, since it will tend to attribute bugs introdu…

Let's start with most outright alarming error - the claude statistics are taken out of whole 2 data points

The interpretations of the p-value is also alarming. One of the first thing they teach you in statistics class is: “an absence of evidence is not evidence of absence”.

This analysis showed that there is indeed an absence of evidence, but it concludes there is evidence of absence.

Traditional p-hacking is done by oversampling and overtesting. If you do 20 analysis on average one will show p p > 0.05

Re: Did Claude increase bugs in rsync?

#174

Post by logicprog »

bakugo wrote: post #169
logicprog wrote: post #160

Earlier quoted context omitted.

[flagged]

Your analysis was so thorough, rigorous, and objective, that you couldn't be bothered to write it yourself. Do you genuinely believe an article written by AI defending itself is going to convince anyone who wasn't already on your side? All you're doing is giving more fuel to the "anti-AI crowd" you hate so much.

Okay, so you didn't respond to any of my rebuttals — like the double standard between anti-AI and pro-AI claims, one of which gets to make claims based on cherry-picked anecdotes, and the other which must produce rigorous studies — you're just going to insult me/my work. Cool.

> Your analysis was so thorough, rigorous, and objective, that you couldn't be bothered to write it yourself. Do you genuinely believe an article written by AI defending itself is going to convince anyone who wasn't already on your side?

Except that I did. I spend days comparing and manually deciding on metrics and methodology – I did not use the AI to decide what I would do or how I would do it, so it is not "the AI defending itself" — then refining things, adding more angles to analyze, and, as I literally say in the opening section, I rewrote all the prose in the entire document just to satisfy critics like you. That sounds like "could be bothered" to me. But people like you will never be satisfied.

Also, even if I hadn't done all that work, that wouldn't make it not rigorous (it clearly is) or objective (it is as objective as it can be with so little data). You're bikeshedding to avoid the point.

Re: Did Claude increase bugs in rsync?

#175

Post by mohamedkoubaa »

thorum wrote: post #80

Unfortunately for the people mad about this, I predict the only thing they will accomplish by pressuring the rsync maintainers, is to discourage everyone else from responsibly disclosing their use of AI. You’re just going to make people disable Claude attribution on their commits to avoid drama.

I'd be willing to be that an undisclosed LLM disclosure will follow a developer around for the rest of their career

Re: Did Claude increase bugs in rsync?

#176

Post by WesolyKubeczek »

The discussions around this have devolved to excrement anyway, I feel tempted to invoke the meme where the goose asking a guy what his jacket is made of, asks “where is your reproducer case!?” instead.

Instead we have a shitstorm over presumably legit issue, for which the only source is some mastodon post.

One command that used to work in 3.4.1 and stopped working in 3.4.3. Just one! We could have already bisected the living shit out of this and go home, but no.

Re: Did Claude increase bugs in rsync?

#177

Post by logicprog »

yobid20 wrote: post #147

needs a tldr; im not reading all that. maybe claude can summarize it for me.

And anti-AI people accuse people who use AI of being intellectually lazy. First of all, it's long because it's expanded to respond to all the criticisms. It seems that either something can be short, and dismissed as incomplete, or it can be complete, and dismissed as being long. Nice Kafka trap. Additionally, there's literally an Executive Summary section right there, for your TLDR.

Re: Did Claude increase bugs in rsync?

#178

Post by logicprog »

MYEUHD wrote: post #170
the_real_cher wrote: post #3

Is there a non vibe coded fork of rsync?

There is openrsync, which the OpenBSD re-implementation of rsync It's not a fork, but it's 8 years old, and is already shipped by default in OpenBSD and macOS.

To quote Tridge:

> As to all the people saying “I’m going to package openrsync for platform XXX and we’ll use that!”. I find that rather amusing. If you do decide to go down that path I’d suggest you try the new rsync test suite on openrsync if you can stomach something that an AI has helped write. I tried it today and openrsync currently fails 85 of 98 tests, so I’m sure it won’t take you long to get it up to speed. You run it like this “./runtests.py — rsync-bin=../openrsync/openrsync — use-tcp”. Admittedly a lot of the failures are just features openrsync doesn’t have, but still, it’s not a great result.

Re: Did Claude increase bugs in rsync?

#179

Post by matheusmoreira »

Groxx wrote: post #168
matheusmoreira wrote: post #162

Earlier quoted context omitted.

> while reviewing As I said, disclosure is polite when contributing code to third party projects which will undergo human review. No need for such things in one's own projects.

> which will undergo human review This can be largely assumed to be true for any open source code. It's kinda the point of open source.

Nope. It cannot be assumed at all. Maintainer could just as easily tell Claude to review the hand written code you sent instead of spending any effort on it. Maintainer could sit on the patch for months on end only to swoop in later and rewrite it instead of engaging with you, thereby erasing your contribution and attribution. Maintainer could just ignore you entirely despite the pervasive "patches welcome" attitude.

If there's one thing I learned not to do in open source, it's to assume nonsense like that.

Re: Did Claude increase bugs in rsync?

#180

Post by steno132 »

This is just narrow thinking. Say Claude did increase the bugs in rsync by a negligible factor.

So what? You've saved a significant amount of time for a decent number of humans, and if those humans are working on other projects, the overall net output for the world is net positive compared to without LLMs.

You have to broaden your perspective. It's not just about how rsync was affected.

Post reply on HN