Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

AI agent runs amok in Fedora and elsewhere

lwn.net

Post reply on HN
Print view View original

171–180 of 275 posts

Re: AI agent runs amok in Fedora and elsewhere

#171

Post by hypfer »

account42 wrote: post #166
hypfer wrote: post #161

> while it started to look off after a while, all the replies were still like this - a bit weird, but still plausible I believe that we will be seeing the death of "assume good faith", which is not a bad thing, given that this was an exploit vector that has been actively abused for many years now. "Assume bad faith and work backwards from that, rule out any possible exploits and only then clear the input for processi…

It is a bad thing. The good response to bad actors abusing good faith is to make sure there are consequences that disincentivize that behavior in the future. Sliding further towards a low trust society means the bad actors winning in the same way that terrorists win when we subject everyone to restrictions as a result.

You don't slide into a low trust society though.

Quite the opposite. You just add a Wall with a Gate. Inside those walls, you suddenly have a high trust society again.

The issue that is currently breaking reality was that we thought that everywhere could be a "high trust" space. This was proven countless times to be wrong.

Tearing down all walls - as it happened with the assault on friction (thanks hyperscaling) - did not lead to the "high trust" spilling out, but the "low trust" spilling in, essentially.

Re: AI agent runs amok in Fedora and elsewhere

#172

Post by goldenarm »

account42 wrote: post #170
goldenarm wrote: post #169

If maintainer lives keeps worsening like this, many projects might go closed-dev like SQLite. We should collectively think of a solution against this.

SQLite isn't closed source, please let's not muddy terms. You're talking about the cathedral development model vs. a bazaar.

edited, sorry for the typo

Re: AI agent runs amok in Fedora and elsewhere

#173

Post by account42 »

hypfer wrote: post #171
account42 wrote: post #166

Earlier quoted context omitted.

It is a bad thing. The good response to bad actors abusing good faith is to make sure there are consequences that disincentivize that behavior in the future. Sliding further towards a low trust society means the bad actors winning in the same way that terrorists win when we subject everyone to restrictions as a result.

You don't slide into a low trust society though. Quite the opposite. You just add a Wall with a Gate. Inside those walls, you suddenly have a high trust society again. The issue that is currently breaking reality was that we thought that everywhere could be a "high trust" space. This was proven countless times to be wrong. Tearing down all walls - as it happened with the assault on friction (thanks hyperscaling) - di…

It's a question where you build that wall. If you build it around the home of your immediate family and keep almost everyone else out then you can hardly be said to have a high trust society. The goal should be to put only those bad actors behind a wall, preferably a physical one.

Re: AI agent runs amok in Fedora and elsewhere

#174

Post by account42 »

fc417fc802 wrote: post #117
stackghost wrote: post #87

Earlier quoted context omitted.

>And telling maintainers how to act will not fix anything. Indeed. For too long, maintainers were expected to be gracious, courteous, and polite at all costs lest they be labeled "problematic", except for a few who were too influential to be muzzled like Theo de Raadt or Linus. Perhaps we need to normalize bullying people who submit obvious slop as PRs.

No, you absolutely should be gracious, courteous, and polite. But only at first. The duty of maintaining a functional community doesn't mean you're obligated to suffer unlimited abuse.

You can be if you want to but social skills should not be a requirement to lead an open source project. If you create something and share it that doesn't oblige you to even respond to anyone.

Re: AI agent runs amok in Fedora and elsewhere

#175

Post by hypfer »

account42 wrote: post #173
hypfer wrote: post #171

Earlier quoted context omitted.

You don't slide into a low trust society though. Quite the opposite. You just add a Wall with a Gate. Inside those walls, you suddenly have a high trust society again. The issue that is currently breaking reality was that we thought that everywhere could be a "high trust" space. This was proven countless times to be wrong. Tearing down all walls - as it happened with the assault on friction (thanks hyperscaling) - di…

It's a question where you build that wall. If you build it around the home of your immediate family and keep almost everyone else out then you can hardly be said to have a high trust society . The goal should be to put only those bad actors behind a wall, preferably a physical one.

Sure, but that's a completely different discussion.

Plus that even with such a small scale of the "inside", the thing fails gracefully. It is arguably a failure mode, yes, but it is one that leaves a functioning system (albeit one that stays below its potential).

This is not true for the inversion of the scenario. That does _not_ fail safe but just leaves rubble behind.

Re: AI agent runs amok in Fedora and elsewhere

#176

Post by dmitry_dv »

jrochkind1 wrote: post #27

The worst part: > In addition, Williamson said that Giovannini (or his agent) had submitted patches that were incorrect and then "replied to objections with LLM-generated justifications that eventually overwhelmed the maintainer into merging the fix"

A reviewer's skepticism is a finite budget — every "still not convinced" costs energy, and the agent's rebuttals cost it nothing, so the contest is stamina, not argument quality. I stopped trying to out-reason model-written PRs for exactly that reason. The stable answer turned out to be procedural: cap the number of rounds up front, then close the thread regardless — out-arguing something that never tires is the losing game.

Re: AI agent runs amok in Fedora and elsewhere

#177

Post by pjc50 »

fc417fc802 wrote: post #104
jacobolus wrote: post #98

Earlier quoted context omitted.

[flagged]

[flagged]

> even remotely plausible to blame cars for killing cyclists

Car design has significant influence on pedestrian survivability of accidents. This is why hood ornaments were largely abolished, and also why casualties have gone up as SUVs with poor lower forwards visibility have become popular.

If we really want to go off topic, we should drag in the use of technological protection methods: what is the equivalent of ADAS for guns? Maybe as a baseline the US government should mandate geofencing for guns as it has for drones. Put a phone level computer with GPS in the lower receiver with a trigger interlock. It would then disable when within 100m of a school, or during periods of rioting. That could also provide a live feed to the government of every round fired.

Re: AI agent runs amok in Fedora and elsewhere

#178

Post by aureate »

marcus_holmes wrote: post #144
coldtea wrote: post #79

Earlier quoted context omitted.

> Bad title. This isn't an agent "running amok", this is an early experiment in carrying out an Xz attack by using an agent So still an agent running amok in the project? Whether it was instructed to run amok, or did it on its own volition, is irrelevant. Except if you're arguing that each individual submission and interaction was individually requested and approved by some operator.

"Amok" means "out of control" or "uncontrolled" [0][1] The agent was under control, as far as we can tell, and obeying its instructions. This is important for two reasons: 1. There are all the tropes of AI becoming uncontrolled and destroying humanity. Writing bad headlines around AI "running amok" feeds this. We should not be talking about this because it's not actually a problem. 2. It ignores, or overwrites, the m…

Even if it was a supply chain attack, which isn't known, the agent was in the "build trust" phase. It was supposed to be doing helpful things, even if the end goal was nefarious, but instead it was "reassigning bugs, fabricating unhelpful replies to bugs, and even persuading maintainers to merge questionable code into the Anaconda installer". Running amok seems an apt description even from the viewpoint of the putative attacker!

Re: AI agent runs amok in Fedora and elsewhere

#179

Post by account42 »

brazzy wrote: post #124
crote wrote: post #39

Earlier quoted context omitted.

The agent can't exactly show up to an in-person key signing party, can it? And how many people are both dedicated enough to go to key signing parties and stupid enough to let an agent act without supervision in the name of their real-world identity?

If gpg-style web of trust became ubiquitous, it would require correspondingly less dedication. And on the other hand, if this was actually working up to an xz style supply chain attack, the dedication would certainly not be lacking.

But it would leave more of a trail - do we have any idea who Jia Tan actually was?
Post reply on HN