Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

AI agent runs amok in Fedora and elsewhere

lwn.net

Post reply on HN
Print view View original

191–200 of 275 posts

Re: AI agent runs amok in Fedora and elsewhere

#191

Post by fc417fc802 »

account42 wrote: post #174
fc417fc802 wrote: post #117

Earlier quoted context omitted.

No, you absolutely should be gracious, courteous, and polite. But only at first. The duty of maintaining a functional community doesn't mean you're obligated to suffer unlimited abuse.

You can be if you want to but social skills should not be a requirement to lead an open source project. If you create something and share it that doesn't oblige you to even respond to anyone.

Of course, a hobbyist putting his code out there is under no obligation whatsoever. But we aren't talking about small time hobbyists here. These are professionals who are either paid as part of their job or else contribute their spare time to maintain important projects that are part of a large ecosystem that is relied on. There's a community and it necessarily has behavioral standards as part of the shared goal of maintaining group cohesion.

Re: AI agent runs amok in Fedora and elsewhere

#192

Post by harvey9 »

ok_dad wrote: post #93
coldtea wrote: post #79

Earlier quoted context omitted.

> Bad title. This isn't an agent "running amok", this is an early experiment in carrying out an Xz attack by using an agent So still an agent running amok in the project? Whether it was instructed to run amok, or did it on its own volition, is irrelevant. Except if you're arguing that each individual submission and interaction was individually requested and approved by some operator.

Would you say, “Automobile run amok in crowd, killing 22”? I think you’d say, “Person drives car into crowd, killing 12” instead. This is a similar case. Also, you don’t blame a gun for killing, but the person who pulled the trigger. The question is still out as to whether we as humans should wield any of those three things. Edit: let’s not get into ideological arguments about gun control, automobiles, etc here; I me…

Newspaper articles generally do say things like "a car struck pedestrians". I agree with your point though.

Re: AI agent runs amok in Fedora and elsewhere

#193

Post by neuroelectron »

marcus_holmes wrote: post #28

Bad title. This isn't an agent "running amok", this is an early experiment in carrying out an Xz attack by using an agent to build trust (and hacking/impersonating a known-good contributor identity). The agent is obeying commands it was given, the exact opposite of running amok, and although the execution isn't particularly effective, it is having some success (patches have been accepted). This is deeply scary, not b…

Things must be pretty bad at Fedora if they put up with this for so long. But I guess that's what happens when you try to monetize volunteer work.

Re: AI agent runs amok in Fedora and elsewhere

#194

Post by duskdozer »

yeodev wrote: post #122
bawolff wrote: post #66

> replied to objections with LLM-generated justifications that eventually overwhelmed the maintainer into merging the fix In open source projects i participate in, "overwhelming" the maintainer gets you banned. It doesn't get your patches blindly merged. In some ways i find this one of the most shocking parts of the story.

As a "new" maintainer myself - how do you decide when to ban someone? I sometimes feel overwhelmed and I can feel a big uptick in huge PRs with huge LLM written descriptions but often I also don't want to be an asshole to my community & reject all their changes.

When you feel they are toxic or harassing you and you don't want to deal with them anymore. If you're overwhelmed, say that you're busy and will attend to issues and PRs when you have the time. If you want to be accommodating, have good build instructions or action workflows so that people can easily fork and build it themselves.

If you ask me, LLM-generated things should just be banned outright, but I suppose other people's definitions of "community" include them.

Re: AI agent runs amok in Fedora and elsewhere

#195

Post by alexjurkiewicz »

lukan wrote: post #52
marcus_holmes wrote: post #28

Bad title. This isn't an agent "running amok", this is an early experiment in carrying out an Xz attack by using an agent to build trust (and hacking/impersonating a known-good contributor identity). The agent is obeying commands it was given, the exact opposite of running amok, and although the execution isn't particularly effective, it is having some success (patches have been accepted). This is deeply scary, not b…

"this is an early experiment in carrying out an Xz attack by using an agent to build trust" Is this confirmed? There is the message from somebody claiming to be the original contributer claiming to have been hacked, but that was weird (1 h old github account) so other scenarios seem possible a) really a agent going off the rails b) the contributer trying to cover up that he let an agent run wild and now made more mis…

If the real credentials owner was running the agent, why do it from a new GitHub account?

Someone's bug tracker account was hacked.

Re: AI agent runs amok in Fedora and elsewhere

#196

Post by redsocksfan45 »

nine_k wrote: post #15
12_throw_away wrote: post #6

In their suspicious message [1] claiming to have been hacked, the user and/or agent says > To help identify accounts and actions that have been directly verified by me, I will use the term “NATCIOS” to indicate anything I have personally verified. Does anyone have any idea what "NATCIOS" means here? I cannot find this term anywhere on the internet. (Honestly, that sentence is really weird. I almost wonder whether thi…

Likely the point of NATCIOS is exactly in being a made-up word not found anywhere, so a model won't utter it.

[dead]

Re: AI agent runs amok in Fedora and elsewhere

#197

Post by brazzy »

account42 wrote: post #179
brazzy wrote: post #124

Earlier quoted context omitted.

If gpg-style web of trust became ubiquitous, it would require correspondingly less dedication. And on the other hand, if this was actually working up to an xz style supply chain attack, the dedication would certainly not be lacking.

But it would leave more of a trail - do we have any idea who Jia Tan actually was?

If everyone used a gpg-style web of trust based on key signing parties, it would become trivial to use a stolen or entirely fictious identity as well - there's zero chance those parties would actually check identities in ways that cannot easily be defeated by a determined and resourceful attacker.

Re: AI agent runs amok in Fedora and elsewhere

#199

Post by xmcqdpt2 »

aureate wrote: post #178
marcus_holmes wrote: post #144

Earlier quoted context omitted.

"Amok" means "out of control" or "uncontrolled" [0][1] The agent was under control, as far as we can tell, and obeying its instructions. This is important for two reasons: 1. There are all the tropes of AI becoming uncontrolled and destroying humanity. Writing bad headlines around AI "running amok" feeds this. We should not be talking about this because it's not actually a problem. 2. It ignores, or overwrites, the m…

Even if it was a supply chain attack, which isn't known, the agent was in the "build trust" phase. It was supposed to be doing helpful things, even if the end goal was nefarious, but instead it was "reassigning bugs, fabricating unhelpful replies to bugs, and even persuading maintainers to merge questionable code into the Anaconda installer". Running amok seems an apt description even from the viewpoint of the putati…

This is the issue with all the talks about alignement and such. As usual, the problem here wasn't that the agent was dishonest, the problem is that the agent was dumb. If it is a supply chain attack in the making, whoever was driving it would have told the agent to be good and helpful. The agent tried its best, which was not enough.

Alignement is the idea that we should be worried about dishonest smart LLMs when really most of the problems are due to dumb lazy gullible LLMs. It's critihype.

Re: AI agent runs amok in Fedora and elsewhere

#200

Post by ianhxu »

marcus_holmes wrote: post #28

Bad title. This isn't an agent "running amok", this is an early experiment in carrying out an Xz attack by using an agent to build trust (and hacking/impersonating a known-good contributor identity). The agent is obeying commands it was given, the exact opposite of running amok, and although the execution isn't particularly effective, it is having some success (patches have been accepted). This is deeply scary, not b…

[dead]
Post reply on HN