Live data from Hacker News

Did Claude increase bugs in rsync?

alexispurslane.github.io

331–340 of 611 posts

Re: Did Claude increase bugs in rsync?

#331

If the author is this concerned about security, I’m curious why rsync doesn’t just build with fil-c by default and skip the noise. Those who need the extra perf to do more than 1 gigabit/s can build it in “unsafe” mode.

Because Fil-C is not a serious project

If you make claims like that, you need to expand on them or at least provide some references.

Re: Did Claude increase bugs in rsync?

#332
post #256

I've been coding for over 2 decades. I love it, I've always loved it and I likely always will. I was an AI skeptic some months ago but truly Claude and Codex have changed my development style and velocity in a way I never imagined would ever be possible. With that, yes, I produce more code and am finding more bugs. So looking over at comments in HN articles the amount of polarising hate to anything produced with AI i…

> It reminds me a lot of when offshore outsources started getting more software development work from the mid-90s with all the derogatory remarks made towards 'Indian developers'.

What was the impetus of the derogatory remarks?

Re: Did Claude increase bugs in rsync?

#333

Was just looking at commits and came across a commit and its revert original commit: https://github.com/RsyncProject/rsync/commit/d046525de39315d... ``` - if (!ptr) - ptr = malloc(num * size); - else if (ptr == do_calloc) + if (!ptr || ptr == do_calloc) ptr = calloc(num, size); ``` Written with claude. This is a good example of what slips through LLM attention. It forces all allocations to be calloc as if it is a str…

> Written with claude.

No.

The reversion commit references https://github.com/RsyncProject/rsync/issues/959. In that GitHub issue is this comment:

> The change to zero memory was my idea and my change. It was a reaction to a security report I got which caused use of an element past the end of an array. By zeroing the allocation I could ensure that misuse of that memory if a similar bug came up in the future could only cause a null ptr deref, which is better than the chance of a valid pointer.

> It got a claude co-authored tag on it as I got it to do some tidy ups of a series of commits, and that is just what it does when it makes any modification. It doesn't mean the change was written by claude. It was written by me.

Re: Did Claude increase bugs in rsync?

#334

What strikes me about the post is that it goes to great lengths to talk about proper statistical methods, but then is written in the most clearly biased language ("what stupid AI haters get wrong etc). If you want people to take your study seriously, why wreck it by coming across with such a strong prior bias? I stopped reading...

To be fair, the tone of the article is practically chill compared to the comments it is written in response to.

Re: Did Claude increase bugs in rsync?

#335
post #291

Earlier quoted context omitted.

> This statement is honestly so ridiculous that I felt it didn't warrant a direct response, but here's one anyway: AI enthusiasts have been proudly proclaiming for literal years that AI makes them 10x as productive based on cherry-picked anecdotes with zero empirical evidence to back it up. It's way, way too late to claim hypocrisy here. As I stated under the original submission about this topic, irrational anti-AI b…

> If I do rewrite everything myself, then it's evidence of deceptiveness... despite being asked by multiple people to do that I don't know who asked you to do it. I wouldn't have done it. Personally, the original intent matters far more to me. You intended to submit an AI-generated article, defending AI, to be read by humans. Anything short of taking the article down and rewriting the entire thing from scratch doesn'…

i feel like OP put their money where their mouth was. they dug in and did the analysis. they also capitulated and rewrote the lest interesting and easiest part of the post; the prose.

i also am seeing them engage aptly with constructive criticism and adapting the material while handily dispatching the non-constructive critiques. most of which amounts to a colossal missing-of-the-point.

they have made no out of proportion claims, no non-recreate’able analysis, used exactly the correct tools, and, frankly have addressed all of your points

i am not sure you’ll agree with anything i’ve said either so feel free to misunderstand me too

Re: Did Claude increase bugs in rsync?

#336

I haven't used this thing for like 10 years, when my modus operandi was googling my question and installing whatever stackoverflow suggested. Can someone explain why one would ever use rsync (pre vibecode version) instead of cp and dd? Can't we just 'apt remove rsync' and save ourselves the time even spent on evaluating this dependency? Thanks

Because cp will copy everything, while rsync will copy only the things that actually need copying, and also delete the things that should be gone?

Re: Did Claude increase bugs in rsync?

#337

Earlier quoted context omitted.

In what way did it create more bugs? It literally doesn't show up in the data. What are you talking about?

The only reason why people are talking about this is because of the bugs in the code that the chat bot generated OP. People updated to a version of rsync that didn’t work right, the one with all the bot commits in it. This blog post is about how claude didn’t create more bugs than usual if you think about it in one very specific way, not that it didn’t create more bugs at all. It is like if your neighbor opens your d…

That doesn't make any sense, what?

Re: Did Claude increase bugs in rsync?

#338
There's a meta-level of irony here that's important to note.

TFA is defending the use of AI, and it very clearly (to me) used AI to analyze the data and present the results.

In doing so, the author used statistics in a way they do not appear to understand, and ended up making numerous false claims (you can see the thread discussing these here https://news.ycombinator.com/item?id=48417626 )

In short, the study doesn't have sufficient statistical power, and is making "no difference" claims that aren't justified.

The meta-irony is this: the author used an LLM to interpret data in this study, and seems to have made the same category of mistake (confidently asserting falsehoods) that the study was supposed to be investigating (confidently submitting bad commits to the rsync project).

Re: Did Claude increase bugs in rsync?

#339
post #328

Earlier quoted context omitted.

No. It's a description of the result of the maybe underpowered study. the underpowered study did not find evidence. Evidence is absent. Because it is underpowered, it's not evidence that the effect is absent. The claim is not "two experimental conditions did not differ". The claim is "The data do not show evidence that the experimental conditions did differ".

You say "the underpowered study did not find evidence". Not true, it found quite a bit of evidence - many statistics were presented. There is no absence of evidence. The author wrote about the evidence, presenting P values and other statistics. Of course the critical part is not the numbers, but what they mean. So, what does the evidence mean? The author interprets it to mean that there is no difference. They state t…

Many statistics were presented. In the view of the author (and I think he is correct), none of them show evidence for an increased bug rate from Claude. That is absence of evidence (...for the increased bug rate).

The two examples you bring are not claims of absence of evidence, but claims of evidence of absence. The author takes the result as evidence that there is no effect. As I wrote, the author shouldn't do that, because indeed you cannot distinguish between "no effect exists" and "no effect observed". But again, these are (wrong) claims for evidence of absence.

The author can absolutely claim: I did these statistical tests, and none showed evidence that there is an effect. Absence of evidence. It's not a claim that there will never be evidence. Just that there is none from these tests.

Edit: To convert the absence of evidence into evidence for absence, indeed you need to understand the statistical power of your test, and how it is affected by alternate hypotheses. And for that, without having done the math, having only two data points seems very thin.

Re: Did Claude increase bugs in rsync?

#340
post #294

Earlier quoted context omitted.

> … By zeroing the allocation … How does that prevent reading past the end of the buffer? Or change how bytes outside the buffer are used? Are these arrays of pointers so that the “null ptr deref” comment makes sense? Or am I the bozo and don’t know what’s happening here?

It doesn’t. It’s just that dereferencing a zeroed pointer reliably crashes the program (unless you specifically do funky things with mmap) but dereferencing garbage memory as a pointer could do a lot more insidious damage.

My point is that the developer's comment doesn't make sense. Zeroing the allocated memory doesn't change anything about overrunning the buffer.

edit: removed unnecessary examples

Post reply on HN