Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

AI agent runs amok in Fedora and elsewhere

lwn.net

Post reply on HN
Print view View original

161–170 of 275 posts

Re: AI agent runs amok in Fedora and elsewhere

#161

Post by hypfer »

> while it started to look off after a while, all the replies were still like this - a bit weird, but still plausible

I believe that we will be seeing the death of "assume good faith", which is not a bad thing, given that this was an exploit vector that has been actively abused for many years now.

"Assume bad faith and work backwards from that, rule out any possible exploits and only then clear the input for processing" will be the new normal.

Which is good. We need friction. Friction makes stuff slow down and work at the speed of humans.

Re: AI agent runs amok in Fedora and elsewhere

#162

Post by account42 »

srdjanr wrote: post #101
ok_dad wrote: post #93

Earlier quoted context omitted.

Would you say, “Automobile run amok in crowd, killing 22”? I think you’d say, “Person drives car into crowd, killing 12” instead. This is a similar case. Also, you don’t blame a gun for killing, but the person who pulled the trigger. The question is still out as to whether we as humans should wield any of those three things. Edit: let’s not get into ideological arguments about gun control, automobiles, etc here; I me…

There's a difference between the driver intentionally driving into crowd, and not intentionally but possibly still recklessly (drifting and losing control, falling asleep, etc). In those cases I would probably use "car hits the crowd", at least in my language

There may be a difference in degree of the crime but the driver is still responsible in both cases and should be the primary subject of any reporting.

Let's reserve "car hits the crowd" for situations where no driver was involved like a break failure on a car parked on a slope or a self-driving car bug.

Re: AI agent runs amok in Fedora and elsewhere

#163

Post by zygentoma »

matsemann wrote: post #127
fc417fc802 wrote: post #112

Earlier quoted context omitted.

> telling maintainers how to act will not fix anything. That depends. In this case it's good actionable advice that should hopefully lower cognitive load. Politely suggest a fork, then if the nagging persists block and move on. Sure if you're in a position of authority you have a responsibility to the community but cutting ties with a stranger who is flagrantly violating social norms is perfectly acceptable. There's…

I don't really think it's actionable. It's like all those campaigns trying to steer behavior, pretty useless. Don't do drugs. Don't speed. Don't drink and drive. You can't just tell people something and expect it to happen. You need systems and guard rails in place. Relying on maintainers to always do the right thing to ensure our security by telling them what to do is not the way.

> It's like all those campaigns trying to steer behavior, pretty useless. Don't do drugs. Don't speed. Don't drink and drive.

They're not useless. They just don't work on the individual level but on the collective. It's a numbers game …

Re: AI agent runs amok in Fedora and elsewhere

#164

Post by account42 »

jdub wrote: post #57
marcus_holmes wrote: post #28

Bad title. This isn't an agent "running amok", this is an early experiment in carrying out an Xz attack by using an agent to build trust (and hacking/impersonating a known-good contributor identity). The agent is obeying commands it was given, the exact opposite of running amok, and although the execution isn't particularly effective, it is having some success (patches have been accepted). This is deeply scary, not b…

I doubt it's that complicated, motivated, or considered... It's probably just garden variety disrespectful behaviour. Purposeless agent spam won't be cheap entertainment forever, but you're right that later stages of industrialised abuse will be scary and unpleasant.

It might not be cheap entertainment forever but it will be cheap cv stuffing for a long time, which has already been a major source of low quality contributions before the aipocalypse.

Re: AI agent runs amok in Fedora and elsewhere

#165

Post by hypfer »

yeodev wrote: post #122
bawolff wrote: post #66

> replied to objections with LLM-generated justifications that eventually overwhelmed the maintainer into merging the fix In open source projects i participate in, "overwhelming" the maintainer gets you banned. It doesn't get your patches blindly merged. In some ways i find this one of the most shocking parts of the story.

As a "new" maintainer myself - how do you decide when to ban someone? I sometimes feel overwhelmed and I can feel a big uptick in huge PRs with huge LLM written descriptions but often I also don't want to be an asshole to my community & reject all their changes.

> I also don't want to be an asshole to my community & reject all their changes.

Do they pay you to triage their noise?

Remember that you owe no one anything at all. Neither legally nor morally. Your chosen license likely even states the former in plain english.

___

Personally, I've adopted the "you annoy me, you're out" stance and have been quite happy with it. You do need a tough shell to do that though as you will be facing all the social exploits people can throw at you.

It also leaves "growth potential" on the table, the same way that limiting your exposure to ionizing radiation does.

That all said, it depends on what your goals are + where in the lifecycle of your project you are. So don't take this as "this is the way" but "this can be one way".

Either way, you're not an asshole for not reading slop. Don't let anyone gaslight you into that.

Re: AI agent runs amok in Fedora and elsewhere

#166

Post by account42 »

hypfer wrote: post #161

> while it started to look off after a while, all the replies were still like this - a bit weird, but still plausible I believe that we will be seeing the death of "assume good faith", which is not a bad thing, given that this was an exploit vector that has been actively abused for many years now. "Assume bad faith and work backwards from that, rule out any possible exploits and only then clear the input for processi…

It is a bad thing. The good response to bad actors abusing good faith is to make sure there are consequences that disincentivize that behavior in the future. Sliding further towards a low trust society means the bad actors winning in the same way that terrorists win when we subject everyone to restrictions as a result.

Re: AI agent runs amok in Fedora and elsewhere

#167

Post by raincole »

Slightly related:

https://x.com/kdaigle/status/2040164759836778878

> There were 1 billion commits in 2025. Now, it's 275 million per week, on pace for 14 billion this year if growth remains linear (spoiler: it won't.)

I think open source as a whole is fucked at this point. No way humans in communities can commit (pun intended) 10x more time to read all of these than before. It'd eventually cost money to submit PR.

Re: AI agent runs amok in Fedora and elsewhere

#168

Post by account42 »

yeodev wrote: post #122
bawolff wrote: post #66

> replied to objections with LLM-generated justifications that eventually overwhelmed the maintainer into merging the fix In open source projects i participate in, "overwhelming" the maintainer gets you banned. It doesn't get your patches blindly merged. In some ways i find this one of the most shocking parts of the story.

As a "new" maintainer myself - how do you decide when to ban someone? I sometimes feel overwhelmed and I can feel a big uptick in huge PRs with huge LLM written descriptions but often I also don't want to be an asshole to my community & reject all their changes.

My solution is to look at PRs and other requests whenever I actually have time and feel like it, prioritizing contributions from people I trust and those that have put in the effort in making my job easier. That might mean things don't get merged for a long time and some people might get upset but that's not my problem.

Re: AI agent runs amok in Fedora and elsewhere

#170

Post by account42 »

goldenarm wrote: post #169

If maintainer lives keeps worsening like this, many projects might go closed-dev like SQLite. We should collectively think of a solution against this.

SQLite isn't closed source, please let's not muddy terms. You're talking about the cathedral development model vs. a bazaar.
Post reply on HN