Live data from Hacker News

Did Claude increase bugs in rsync?

alexispurslane.github.io

211–220 of 611 posts

Re: Did Claude increase bugs in rsync?

#211
post #80

Unfortunately for the people mad about this, I predict the only thing they will accomplish by pressuring the rsync maintainers, is to discourage everyone else from responsibly disclosing their use of AI. You’re just going to make people disable Claude attribution on their commits to avoid drama.

“Don’t get mad at people for doing something unethical or immoral, or they’ll do something unethical or immoral!” Disabling attribution of LLM-generated code is fraud, because you’re saying you wrote the code. Of course that fits right in with the use of an LLM to generate code in the first place, since what it’s actually doing is regurgitating its inputs stripped of any license and copyright notice.

I'm very certain that this is not fraud, across multiple legal systems, both roman and common law. In both cases fraud requires a person is deprived of a material good. Neither the defrauded person or their material loss is present in this case. Maybe there is a oddball legal system somewhere in the world where fraud is something entirely different, but i doubt it. "Fraud", just like "Decorator Pattern" is a well established concept and pretty simple concept, even if there are edge cases. This does not fit at all.

In academia this is miss-attribution, outside of academia this does not exist.

This is clearly not not copyright infringement either as LLMs do not claim copyright, nor could they. Just like the photograph taken by the monkey, or pictures drawn by crows. LLM output is not a creative work either.

If this is unethical or immoral is a totaly different question. I really dont think so and I dont think you argue that position well.

Re: Did Claude increase bugs in rsync?

#212

Earlier quoted context omitted.

Let's start with most outright alarming error - the claude statistics are taken out of whole 2 data points

The interpretations of the p-value is also alarming. One of the first thing they teach you in statistics class is: “an absence of evidence is not evidence of absence” . This analysis showed that there is indeed an absence of evidence, but it concludes there is evidence of absence. Traditional p-hacking is done by oversampling and overtesting. If you do 20 analysis on average one will show p p > 0.05

The concept you need here is "Statistical Power".

The ELI5 version is that there are two mistakes you can make when looking at a P value:

Type I error, where your P value is falsely low. In the experiment being discussed here, it would lead one to conclude that AI code is worse. Otherwise known as a false positive.

Type II error, where your P value is falsely high, leading you to conclude that AI code is no different. Otherwise known as a false negative.

https://en.wikipedia.org/wiki/Power_(statistics)

One can calculate statistical power for a given experimental protocol.

My hunch is that if you did this, you would find this experiment is grossly under-powered.

This means you can't make the "absence of evidence" claim.

Re: Did Claude increase bugs in rsync?

#213

Earlier quoted context omitted.

It makes no sense at all to do that. The only thing that matters is whether the code is good.

That’s not the only thing that matters. The provenance of the code also matters enormously, specifically whether the person contributing it actually has the right to do so. If I contributed code to an Open Source project behind my old employer’s back, that would have been bad, because that code was owned by them and not me, even if I wrote it on my own time using my own equipment, because of the contract I signed wit…

[deleted]

Re: Did Claude increase bugs in rsync?

#214

Earlier quoted context omitted.

It makes no sense at all to do that. The only thing that matters is whether the code is good.

That’s not the only thing that matters. The provenance of the code also matters enormously, specifically whether the person contributing it actually has the right to do so. If I contributed code to an Open Source project behind my old employer’s back, that would have been bad, because that code was owned by them and not me, even if I wrote it on my own time using my own equipment, because of the contract I signed wit…

"LLM produced licensed code and person contributed it" is indistinguishable from "person contributed licensed code". The LLM is irrelevant. Result is the same as if they had copy pasted it.

Re: Did Claude increase bugs in rsync?

#215

Earlier quoted context omitted.

> This analysis showed that there is indeed an absence of evidence, but it concludes there is evidence of absence. I tried pretty hard to avoid saying that, can you point me at how to rephrase? The point I'm trying to make is just that there is absolutely no evidence at all for what people are saying with such absolutism and claimed objectivity (that Claude made rsync worse), and thus it doesn't justify the outrage.…

You are interpreting the p-values on their own merit rather then using them to test a null-hypothesis. Quotes like: > With a p-value of 74%, the answer is a decisive no. The odds ratio is 1.06 — essentially 1:1. Claude releases are no more likely to be above the median than any other releases. are problematic in this context as the correct conclusion here is you just don‘t have enough data conclude whether or not you…

Edited that claim, and made several clarifications elsewhere. The whole point of this analysis is that outrage is unjustified on the basis of two totally statistically unremarkable releases that no one would have remarked on pre-AI (my further proof of this is that there was a pre-AI remarkably broken release, and no one did comment!) and zero positive evidence outside cherry-picked anecdotes for any negative impact. We should wait for outrage and version pinning and cancelation until there is evidence, no? I'm just trying to say that these specific releases are unremarkable, and there's no evidence at all of harm currently; I'm not trying to build any kind of predictive model for future Claude releases to say anything grander than "these specific releases are fine, what are we freaking out about?", not some claim about what Claude-exposed releases will look like or trend like in the future or in general.

Re: Did Claude increase bugs in rsync?

#216

Earlier quoted context omitted.

It makes no sense at all to do that. The only thing that matters is whether the code is good.

That’s not the only thing that matters. The provenance of the code also matters enormously, specifically whether the person contributing it actually has the right to do so. If I contributed code to an Open Source project behind my old employer’s back, that would have been bad, because that code was owned by them and not me, even if I wrote it on my own time using my own equipment, because of the contract I signed wit…

[flagged]

Re: Did Claude increase bugs in rsync?

#218
post #203

Earlier quoted context omitted.

Okay, so you didn't respond to any of my rebuttals — like the double standard between anti-AI and pro-AI claims, one of which gets to make claims based on cherry-picked anecdotes, and the other which must produce rigorous studies — you're just going to insult me/my work. Cool. > Your analysis was so thorough, rigorous, and objective, that you couldn't be bothered to write it yourself. Do you genuinely believe an arti…

> like the double standard between anti-AI and pro-AI claims, one of which gets to make claims based on cherry-picked anecdotes, and the other which must produce rigorous studies This statement is honestly so ridiculous that I felt it didn't warrant a direct response, but here's one anyway: AI enthusiasts have been proudly proclaiming for literal years that AI makes them 10x as productive based on cherry-picked anecd…

> This statement is honestly so ridiculous that I felt it didn't warrant a direct response, but here's one anyway: AI enthusiasts have been proudly proclaiming for literal years that AI makes them 10x as productive based on cherry-picked anecdotes with zero empirical evidence to back it up. It's way, way too late to claim hypocrisy here. As I stated under the original submission about this topic, irrational anti-AI behavior is usually just an equal and opposite reaction to irrational pro-AI behavior.

I'm talking about the double standard on the anti-AI side about what evidence should count, not some vague industry-wide epistemic standard, whatever that means. I'm aware LinkedIn Lunatics and Steve Yegge are also being crazy. And it seems to me that even your response here is engaging in a bit of a double standard, or something akin to it, in that you think the irrational anti-AI behavior should be given a pass — and the conclusions perhaps even taken seriously — just because pro-AI people did it too.

> And that doesn't help. If anything, editing the AI output to make it read less like blatant slop just comes off as deceptive, like you're trying to hide the fact that the analysis was AI generated.

Okay, so, if I don't spend the time to write everything myself, that's bad because it's AI slop. If I do rewrite everything myself, then it's evidence of deceptiveness... despite being asked by multiple people to do that, and being extremely explicit about my methods and process and the commit history being (as you've shown), very public.

Also, the AI-generatedness of the text doesn't mean the analysis is AI generated, in terms of what was actually done. That's a category error.

> Looking at the commits, you were adding more AI generated text less than 2 hours ago[0] before quickly editing out one of the most blatantly sloppy sentences I've ever read[1].

The second commit literally says that that was my prose it was fucking with by adding slop. It's just that me adding my prose, and it adding slop to it, were in the same previous commit. Additionally, my process is often giving it exactly what I want to say, more or less, and having it HTML-format it and insert the templated numbers and UI widgets around that text.

But again, even if I'm spending the time to read through and edit everything it's writing to de-slop it, then I'm clearly also reading it through enough to make sure the analysis makes sense, and is accurate; how is that not enough "effort" for you, if effort is supposed to be a proxy for verification?

> Even if we ignore the bias clearly on display there, the premise alone is enough to dismiss the entire thing as heavily biased and chasing a pre-determined conclusion - of course someone who is so dependent and trustful of AI that they decide such an analysis on the bugginess of AI code should itself be written by AI is going to steer the conclusion towards "actually AI code is good and you luddites are overreacting".

That's not ignoring the bias, that's literally restating that you think the bias is there. But if you really think that my bias meaningfully "steered the results," then show me how that happened. Tell me how you would've proven the Claude releases were meaningfully worse, or unusual, at all, or how the methods I chose biased the data against that result, or literally anything except shifting the goalposts and using accusations of "bias" as a get-out-of-jail-free-card.

> The entire concept is so tone-deaf that failing to notice it or predict the criticism before publishing is enough to prove the bias.

And you're so committed to your preconceived notions that anything made with AI must be bad, wrong, or not worth your time, that you'll spend your entire time begging the question ("it's made with AI, therefore it's wrong") and shifting the goalposts instead of engaging meaningfully.

Also, I certainly predicted the criticism (in general, anyway, to the fact that it was made with AI; not the prose being AI) but I made it this way anyway, because if someone is so AI-blinded that they can't read and evaluate the actual metrics, methodology, and provide meaningful criticism to it, and instead can only see that it was made with AI, and they're so it doesn't matter.

Nothing you have said makes the analysis wrong. At this point, you're essentially just resorting to ad homenem and begging the question.

Re: Did Claude increase bugs in rsync?

#219
It's always the most insufferable people that make the biggest hullabaloo about a project they have nothing to do with and have never contributed to. People with literally zero skin in the game using the AI boogeyman to push some agenda or some anti-agenda. OSS has become so incredibly toxic in the past decade, and consumers of OSS have become extremely entitled.

I run a smallish project with ~1k stars and I've stopped maintaining it last year because people feel like they're absolutely owed features or bug-fixes or whatever. It's tiring and a complete shame that author has to make such an insane deep dive into a random accusation that just caught on social media. I want to emphasize that this has nothing to do with AI, it's just tech tourists, consumers (as opposed to creators), and engagement farmers that have taken over. AI slop probably doesn't help, but the underlying issue has been brewing for at least a decade.

Also, the "making soup for the homeless & pissing in it" is not only an off-base analogy (software is pretty low on Maslow’s Hierarchy of Needs), but also somehow looks down on both people in need and the volunteers that help them. Just absolutely gross.

Post reply on HN