Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

AI agent runs amok in Fedora and elsewhere

lwn.net

Post reply on HN
Print view View original

201–210 of 275 posts

Re: AI agent runs amok in Fedora and elsewhere

#201

Post by FeepingCreature »

marcus_holmes wrote: post #144
coldtea wrote: post #79

Earlier quoted context omitted.

> Bad title. This isn't an agent "running amok", this is an early experiment in carrying out an Xz attack by using an agent So still an agent running amok in the project? Whether it was instructed to run amok, or did it on its own volition, is irrelevant. Except if you're arguing that each individual submission and interaction was individually requested and approved by some operator.

"Amok" means "out of control" or "uncontrolled" [0][1] The agent was under control, as far as we can tell, and obeying its instructions. This is important for two reasons: 1. There are all the tropes of AI becoming uncontrolled and destroying humanity. Writing bad headlines around AI "running amok" feeds this. We should not be talking about this because it's not actually a problem. 2. It ignores, or overwrites, the m…

> 1. There are all the tropes of AI becoming uncontrolled and destroying humanity. Writing bad headlines around AI "running amok" feeds this. We should not be talking about this because it's not actually a problem.

if humanity gets destroyed by AI obeying its instructions I'm sure everyone will be very relieved that we didn't pay any attention to fake made up problems like AI not obeying instructions, which of course never happens.

Re: AI agent runs amok in Fedora and elsewhere

#202

Post by wmanley »

lionkor wrote: post #148
yeodev wrote: post #122

Earlier quoted context omitted.

As a "new" maintainer myself - how do you decide when to ban someone? I sometimes feel overwhelmed and I can feel a big uptick in huge PRs with huge LLM written descriptions but often I also don't want to be an asshole to my community & reject all their changes.

Remove the human element. Yes, someone spent time fixing a bug. If the fix doesn't look like it makes sense on its own, do not merge it. If the author tries to convince you that it's a good fix, it's an immediate no. A good fix (which is the only acceptable fix in open-source software), is one that speaks for itself.

> A good fix (which is the only acceptable fix in open-source software), is one that speaks for itself.

I disagree. Often if I'm making a PR to an open-source project I'm doing so because I have a use-case that the original author hadn't considered. So the first step in getting the PR merged is explaining my point of view and convincing the maintainer that my use-case is valid. Only when this is done can the "goodness" of the patch be evaluated.

Re: AI agent runs amok in Fedora and elsewhere

#203

Post by [deleted] »

yeodev wrote: post #122
bawolff wrote: post #66

> replied to objections with LLM-generated justifications that eventually overwhelmed the maintainer into merging the fix In open source projects i participate in, "overwhelming" the maintainer gets you banned. It doesn't get your patches blindly merged. In some ways i find this one of the most shocking parts of the story.

As a "new" maintainer myself - how do you decide when to ban someone? I sometimes feel overwhelmed and I can feel a big uptick in huge PRs with huge LLM written descriptions but often I also don't want to be an asshole to my community & reject all their changes.

[deleted]

Re: AI agent runs amok in Fedora and elsewhere

#204

Post by RetroTechie »

Forgeties79 wrote: post #78
hn773746483 wrote: post #31

Earlier quoted context omitted.

It's just social engineering. No different than say, 2FA fatigue (blowing up someone's phone with 2FA "is this you? yes/no" prompts until user/child/wife/SO/etc clicks yes) or even just simply harassing IT helpdesk until they reset "your" password.

“Before LLM’s there was_____” I see this whenever an LLM’s impact is assessed. We know. The issue is scale and the ability for smaller and smaller groups (down to individuals) to execute at scale. LLM’s are pouring massive amount of gasoline on existing issues and people just keep shrugging. Fake news always existed. Now one dude in India can flood multiple sock puppet media accounts with right wing content/images (a…

> LLM’s are pouring massive amount of gasoline on existing issues and people just keep shrugging.

To use your analogy: this is much like a forest fire. Tinder-dry combustible stuff is piled up everywhere, there's no lack of ignition sources, and firefighters are thin on the ground.

Fun times ahead.

Re: AI agent runs amok in Fedora and elsewhere

#205

Post by m4rtink »

keyle wrote: post #9

There is a natural pace of humans requiring food, water and sleep. The main issue with suspicious AI agents is that they never sleep. So it will take extra-coordination between timezones to ensure we don't let them in. Fundamentally, until we can really prove we're humans online, open-source has a real problem on its hands. Contributions from people from identities known and consistent before the AI-age are fine, eve…

Not to mention people who are still on the other side nominally in control but send LLM generated patches without declaring them as such.

Then you basically need to review any review from people that might be long term contributors but you don't know personally as new contributor patches, as the code is not from their head & you can't risk them properly reviewing it on their end.

To a degree its will always be a new contributor - an amnesiac LLM prompted to produce the patch with zero memory of any past PRs & lot of entropy in the mix.

Re: AI agent runs amok in Fedora and elsewhere

#206

Post by m4rtink »

Leonard_of_Q wrote: post #76

There's a clear solution to the danger posed to free software projects by accepting hostile submissions but it probably is not one that maintainers want to hear: they can use an agent to check submissions for nefarious patterns. Sometimes you fight fire with fire.

So next the attacker puts prompt injection in their PRs & take control of the agent on your end. Perfect, 10 out of 10.

Re: AI agent runs amok in Fedora and elsewhere

#207

Post by frumiousirc »

yeodev wrote: post #122
bawolff wrote: post #66

> replied to objections with LLM-generated justifications that eventually overwhelmed the maintainer into merging the fix In open source projects i participate in, "overwhelming" the maintainer gets you banned. It doesn't get your patches blindly merged. In some ways i find this one of the most shocking parts of the story.

As a "new" maintainer myself - how do you decide when to ban someone? I sometimes feel overwhelmed and I can feel a big uptick in huge PRs with huge LLM written descriptions but often I also don't want to be an asshole to my community & reject all their changes.

I think everyone / every project needs to adopt a strategy consistent with their values.

Unfortunately, I see the choice space here as having "developer effort" anti-correlated with "negative repercussions".

On one end of the distribution, a "hair trigger ban" strategy is low-effort for the developer but will have some fraction of false positives and some fraction of those impacted will complain to "the socials" and some fraction of those complaints will gain traction and, as we have seen, can unfairly taint the project or worse. Responding and managing the false positives also requires developer effort, unless the developers can sustain a "fsck the haters" attitude.

On the other end of the distribution, the developer can spends substantial effort to engage each submitter to ascertain and correct bad behavior, educate them on how they should engage other humans as a fellow human in this LLM era.

There is developer effort needed of different types along this distribution.

A divide-and-conquer strategy might go something like this:

- Rank each submission in some low dimension space (llmhuman, malicioushelpful)

- When enough samples are collected, perform clustering in this space to determine stereotypes, name these clusters, and develop mitigating strategies and implementations as needed.

Mitigations from easy/extreme to hard/accommodating could include:

- Hair trigger ban button.

- Copy-paste a link to an explanation in a comment before closing and/or banning.

- Customized explanation in comment before closing and/or banning.

- Link or customized explanation of what must be done to move the sample to a more favorable category and close/ban if resistance or silence is returned.

- Ongoing engagement in the face of resistance or silence.

This "meta development" program to provide such a system/facility could of course be highly automated with LLMs, fighting fire with fire.

(Despite the length of this reply, it was written entirely by a random human on the internet and not an LLM).

Re: AI agent runs amok in Fedora and elsewhere

#208

Post by m4rtink »

dbdbdbdbdb wrote: post #69

The even more scary thought is if the part owning the ai, that everyone uses, is controlled by someone with different agenda. Say a state actor. What an easy way for that actor to introduce backdoors all over the place or to take over any developers laptop that it want to target. How can anyone trust these tools and how can anyone not use them since they give so much value. I've been programming my whole life and bee…

Yeah, I am quite surprised this is not discussed more often - for remote cloud based AI not only does the provider see everything you provide to the tool/agent, there is no guarantee they can't manipulate the output at any time for a direct attack or more malicious purpose (fetch keys/secrets, put malware in place).

Even with locally running models this can't be singled out given how blackbox models generated by others are. You would have to generate the model yourself from clean data to be reasonably safe.

Re: AI agent runs amok in Fedora and elsewhere

#209

Post by brookst »

FeepingCreature wrote: post #201
marcus_holmes wrote: post #144

Earlier quoted context omitted.

"Amok" means "out of control" or "uncontrolled" [0][1] The agent was under control, as far as we can tell, and obeying its instructions. This is important for two reasons: 1. There are all the tropes of AI becoming uncontrolled and destroying humanity. Writing bad headlines around AI "running amok" feeds this. We should not be talking about this because it's not actually a problem. 2. It ignores, or overwrites, the m…

> 1. There are all the tropes of AI becoming uncontrolled and destroying humanity. Writing bad headlines around AI "running amok" feeds this. We should not be talking about this because it's not actually a problem. if humanity gets destroyed by AI obeying its instructions I'm sure everyone will be very relieved that we didn't pay any attention to fake made up problems like AI not obeying instructions, which of course…

Are you suggesting we should embrace imprecise / false use of language because the vibes are right?

That seems a “part of the problem” move to me. If we can’t be bothered to get things right, how are we better than runamok AI?

Post reply on HN