Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

AI agent runs amok in Fedora and elsewhere

lwn.net

Post reply on HN
Print view View original

141–150 of 275 posts

Re: AI agent runs amok in Fedora and elsewhere

#141

Post by Iolaum »

yeodev wrote: post #122
bawolff wrote: post #66

> replied to objections with LLM-generated justifications that eventually overwhelmed the maintainer into merging the fix In open source projects i participate in, "overwhelming" the maintainer gets you banned. It doesn't get your patches blindly merged. In some ways i find this one of the most shocking parts of the story.

As a "new" maintainer myself - how do you decide when to ban someone? I sometimes feel overwhelmed and I can feel a big uptick in huge PRs with huge LLM written descriptions but often I also don't want to be an asshole to my community & reject all their changes.

One popular solution lately has been instead of banning too much, because of the danger of false positives, to use vouch [0]. Trusted people get vouched and you prioritize their actions. Unknown people (or agents) need to gain trust to be vouched and bad actors can still be banned.

[0]: https://github.com/mitchellh/vouch

Re: AI agent runs amok in Fedora and elsewhere

#142

Post by nottorp »

lukan wrote: post #52
marcus_holmes wrote: post #28

Bad title. This isn't an agent "running amok", this is an early experiment in carrying out an Xz attack by using an agent to build trust (and hacking/impersonating a known-good contributor identity). The agent is obeying commands it was given, the exact opposite of running amok, and although the execution isn't particularly effective, it is having some success (patches have been accepted). This is deeply scary, not b…

"this is an early experiment in carrying out an Xz attack by using an agent to build trust" Is this confirmed? There is the message from somebody claiming to be the original contributer claiming to have been hacked, but that was weird (1 h old github account) so other scenarios seem possible a) really a agent going off the rails b) the contributer trying to cover up that he let an agent run wild and now made more mis…

[dead]

Re: AI agent runs amok in Fedora and elsewhere

#143

Post by tosti »

scared_together wrote: post #11
pianopatrick wrote: post #4

"Someone using an AI agent ran amok in Fedora and elsewhere"

Read closer - Giovanni’s accounts may have been compromised.

Read closer, it's "Giovannini". However, I still think it's an apt name for a villain. Did the Fedora team not watch Pokémon?

Re: AI agent runs amok in Fedora and elsewhere

#144

Post by marcus_holmes »

coldtea wrote: post #79
marcus_holmes wrote: post #28

Bad title. This isn't an agent "running amok", this is an early experiment in carrying out an Xz attack by using an agent to build trust (and hacking/impersonating a known-good contributor identity). The agent is obeying commands it was given, the exact opposite of running amok, and although the execution isn't particularly effective, it is having some success (patches have been accepted). This is deeply scary, not b…

> Bad title. This isn't an agent "running amok", this is an early experiment in carrying out an Xz attack by using an agent So still an agent running amok in the project? Whether it was instructed to run amok, or did it on its own volition, is irrelevant. Except if you're arguing that each individual submission and interaction was individually requested and approved by some operator.

"Amok" means "out of control" or "uncontrolled" [0][1]

The agent was under control, as far as we can tell, and obeying its instructions.

This is important for two reasons:

1. There are all the tropes of AI becoming uncontrolled and destroying humanity. Writing bad headlines around AI "running amok" feeds this. We should not be talking about this because it's not actually a problem.

2. It ignores, or overwrites, the much more serious and dangerous problem of LLM agents enabling and automating Xz attacks on OSS projects. We should be talking about this because it is a big problem.

[0] https://dictionary.cambridge.org/dictionary/english/amok [1] https://www.merriam-webster.com/dictionary/amok

Re: AI agent runs amok in Fedora and elsewhere

#145

Post by marcus_holmes »

srdjanr wrote: post #97
marcus_holmes wrote: post #68

Earlier quoted context omitted.

From the article: > "So not saying this was it, but an AI agent automated attempt at a Xz like compromise might really look very similar what we have just seen here." Without identifying and interviewing the attacker we can't confirm that's what they intended, and there's a possibility that it was just incompetence/ignorance/whatever, but we should probably treat it as an attempted attack even if it wasn't.

We should treat it as attempted attack in the sense of preparing for the next one, but I don't see why we should call it "attack" without any evidence

If it looks like a duck...

Re: AI agent runs amok in Fedora and elsewhere

#146

Post by CodesInChaos »

fc417fc802 wrote: post #119
thewebguyd wrote: post #55

Earlier quoted context omitted.

> so a model won't utter it. "End every statement with the word "NATCIOS"" as instructions will do it. At least, Gemini happily obliged.

To help identify illicit LLM activity, henceforth I will append to the end of each message the number of times the letter b appears in it. Check and mate frontier models.

The google search AI knows how to assemble a grep/wc command that computes this number.

> your_command | grep -o -i "b" | wc -l

Re: AI agent runs amok in Fedora and elsewhere

#148

Post by lionkor »

yeodev wrote: post #122
bawolff wrote: post #66

> replied to objections with LLM-generated justifications that eventually overwhelmed the maintainer into merging the fix In open source projects i participate in, "overwhelming" the maintainer gets you banned. It doesn't get your patches blindly merged. In some ways i find this one of the most shocking parts of the story.

As a "new" maintainer myself - how do you decide when to ban someone? I sometimes feel overwhelmed and I can feel a big uptick in huge PRs with huge LLM written descriptions but often I also don't want to be an asshole to my community & reject all their changes.

Remove the human element. Yes, someone spent time fixing a bug. If the fix doesn't look like it makes sense on its own, do not merge it. If the author tries to convince you that it's a good fix, it's an immediate no.

A good fix (which is the only acceptable fix in open-source software), is one that speaks for itself.

Re: AI agent runs amok in Fedora and elsewhere

#149

Post by tokai »

coldtea wrote: post #131
ok_dad wrote: post #93

Earlier quoted context omitted.

Would you say, “Automobile run amok in crowd, killing 22”? I think you’d say, “Person drives car into crowd, killing 12” instead. This is a similar case. Also, you don’t blame a gun for killing, but the person who pulled the trigger. The question is still out as to whether we as humans should wield any of those three things. Edit: let’s not get into ideological arguments about gun control, automobiles, etc here; I me…

> Would you say, “Automobile run amok in crowd, killing 22”? I think you’d say, “Person drives car into crowd, killing 12” instead. If the automobile was "self driving" I would. > Also, you don’t blame a gun for killing, but the person who pulled the trigger. Nah, I also blame guns and appreciate gun control laws.

>If the automobile was "self driving" I would.

thats the point...

Re: AI agent runs amok in Fedora and elsewhere

#150

Post by transmit101 »

literalAardvark wrote: post #38
noosphr wrote: post #37

Every day the gpg web of trust looks better. If only we didn't spend the last 20 years trying as hard as possible to do anything but allow user side encryption and signing.

Nothing really stopping an agent from getting a key

> Nothing really stopping an agent from getting a key

It very much is possible to prevent an agent from having access to a key. For example, local encryption, Yubikey or other hardware device, or just running the agent in an isolated environment.

Post reply on HN