Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

AI agent runs amok in Fedora and elsewhere

lwn.net

Post reply on HN
Print view View original

211–220 of 275 posts

Re: AI agent runs amok in Fedora and elsewhere

#211

Post by m4rtink »

account42 wrote: post #173
hypfer wrote: post #171

Earlier quoted context omitted.

You don't slide into a low trust society though. Quite the opposite. You just add a Wall with a Gate. Inside those walls, you suddenly have a high trust society again. The issue that is currently breaking reality was that we thought that everywhere could be a "high trust" space. This was proven countless times to be wrong. Tearing down all walls - as it happened with the assault on friction (thanks hyperscaling) - di…

It's a question where you build that wall. If you build it around the home of your immediate family and keep almost everyone else out then you can hardly be said to have a high trust society . The goal should be to put only those bad actors behind a wall, preferably a physical one.

Yeah, gated communities like that are usually a clear sign that something bad is happening with the given society - or in a minor cases with the community, if it needs to gate itself from a society that is not failing.

Re: AI agent runs amok in Fedora and elsewhere

#212

Post by brookst »

xmcqdpt2 wrote: post #199
aureate wrote: post #178

Earlier quoted context omitted.

Even if it was a supply chain attack, which isn't known, the agent was in the "build trust" phase. It was supposed to be doing helpful things, even if the end goal was nefarious, but instead it was "reassigning bugs, fabricating unhelpful replies to bugs, and even persuading maintainers to merge questionable code into the Anaconda installer". Running amok seems an apt description even from the viewpoint of the putati…

This is the issue with all the talks about alignement and such. As usual, the problem here wasn't that the agent was dishonest, the problem is that the agent was dumb. If it is a supply chain attack in the making, whoever was driving it would have told the agent to be good and helpful. The agent tried its best, which was not enough. Alignement is the idea that we should be worried about dishonest smart LLMs when real…

“Be good and helpful” is one possible instruction, but it’s a leap to think it’s the only possible one.

Perhaps there was an automated harness that was intended to be good and helpful for a year, but a bug caused it to flip to malicious too quickly.

Or perhaps it was intentional, to test the behavior, and they just didn’t care about discovery here.

Or…

Though I am in agreement that a lot of issues in this space come from lazy, gullible actors.

Re: AI agent runs amok in Fedora and elsewhere

#213

Post by m4rtink »

alexjurkiewicz wrote: post #195
lukan wrote: post #52

Earlier quoted context omitted.

"this is an early experiment in carrying out an Xz attack by using an agent to build trust" Is this confirmed? There is the message from somebody claiming to be the original contributer claiming to have been hacked, but that was weird (1 h old github account) so other scenarios seem possible a) really a agent going off the rails b) the contributer trying to cover up that he let an agent run wild and now made more mis…

If the real credentials owner was running the agent, why do it from a new GitHub account? Someone's bug tracker account was hacked.

So far it looks like just their previously legit Fedora account got taken over & the other accounts (GitHub) then generated on demand as needed for whatever it was trying to achieve, right ?

BTW, any idea what are the current requirements for creating a new GitHub account ? That could provide some information about if there was actually a person controlling thing thing at that moment to say provide wahtever was necessary to get the new GitHub account.

Re: AI agent runs amok in Fedora and elsewhere

#214

Post by brookst »

fc417fc802 wrote: post #135
coldtea wrote: post #132

Earlier quoted context omitted.

A phrase like "who hasn't been driven mad by propaganda" doesn't exactly sound like impationately discussing the issue either.

Calling a zealot a zealot does not mean that one is incapable of discussing the underlying topic. We must not let the desire to converse intelligently hamstring our ability to call out obviously corrupt patterns of thought for what they are. Anyway my above reply was hardly the appropriate venue to engage in a genuine manner on that topic. The parent was blatantly derailing things by inserting his pet political issue…

I agree, and I also agree that zealots who cast anyone who disagrees with them as being literally insane should also not be indulged.

Re: AI agent runs amok in Fedora and elsewhere

#215

Post by RetroTechie »

duskdozer wrote: post #194
yeodev wrote: post #122

Earlier quoted context omitted.

As a "new" maintainer myself - how do you decide when to ban someone? I sometimes feel overwhelmed and I can feel a big uptick in huge PRs with huge LLM written descriptions but often I also don't want to be an asshole to my community & reject all their changes.

When you feel they are toxic or harassing you and you don't want to deal with them anymore. If you're overwhelmed, say that you're busy and will attend to issues and PRs when you have the time. If you want to be accommodating, have good build instructions or action workflows so that people can easily fork and build it themselves. If you ask me, LLM-generated things should just be banned outright, but I suppose other…

> If you ask me, LLM-generated things should just be banned outright,

Why? In the end it's a patch's quality that counts. Regardless who or what contributed it.

Bad patch from trusted contributor is still a bad patch.

Perhaps this is more a management problem. How to best use developer's time, where to use AI (vs blindly deploy AI to generate patches & swamp developers with that).

Or do some rate-limiting? "Sorry, we accept no more than 10KB worth of patches per week on this project! Try again next week after we've reviewed this week's batch".

Re: AI agent runs amok in Fedora and elsewhere

#216

Post by voidUpdate »

LoganDark wrote: post #64
jaypatelani wrote: post #56

Earlier quoted context omitted.

That's some of the reasons NetBSD don't accept LLM/AI tainted code

I am sad people conflate this stuff with LLMs being bad. You can condemn the bad behavior without banning an entire technology.

Yeah, LLMs are bad for a whole different set of reasons than they write bad code

Re: AI agent runs amok in Fedora and elsewhere

#217

Post by comboy »

jdub wrote: post #57
marcus_holmes wrote: post #28

Bad title. This isn't an agent "running amok", this is an early experiment in carrying out an Xz attack by using an agent to build trust (and hacking/impersonating a known-good contributor identity). The agent is obeying commands it was given, the exact opposite of running amok, and although the execution isn't particularly effective, it is having some success (patches have been accepted). This is deeply scary, not b…

I doubt it's that complicated, motivated, or considered... It's probably just garden variety disrespectful behaviour. Purposeless agent spam won't be cheap entertainment forever, but you're right that later stages of industrialised abuse will be scary and unpleasant.

Here's the thing. Building trust and then leaving stuff in has been around forever. The fact that it becomes cheaper does not matter that much (since protection against it is also getting better), but it required you to have a bunch of extremely talented people who has spent much of their life diving into given topic.

Such driven people are usually even hard to buy, they usually would rather get by with enough income and work on interesting projects with interesting people that get some uninteresting work for tons of money. This still does not stop them from working for Malice. But ethics do. Even if not right away, if people see that what they are doing is not quite OK, the talent stops eroding. People quit, productivity drops. That was a good dynamic. Which now will be gone.

Re: AI agent runs amok in Fedora and elsewhere

#218

Post by lionkor »

wmanley wrote: post #202
lionkor wrote: post #148

Earlier quoted context omitted.

Remove the human element. Yes, someone spent time fixing a bug. If the fix doesn't look like it makes sense on its own, do not merge it. If the author tries to convince you that it's a good fix, it's an immediate no. A good fix (which is the only acceptable fix in open-source software), is one that speaks for itself.

> A good fix (which is the only acceptable fix in open-source software), is one that speaks for itself. I disagree. Often if I'm making a PR to an open-source project I'm doing so because I have a use-case that the original author hadn't considered. So the first step in getting the PR merged is explaining my point of view and convincing the maintainer that my use-case is valid. Only when this is done can the "goodnes…

It's usually better to create an issue where you explain this, then the PR is just the change. But this is up to each maintainer to decide, I guess.

Re: AI agent runs amok in Fedora and elsewhere

#219

Post by amenhotep »

PhilipRoman wrote: post #128
ok_dad wrote: post #93

Earlier quoted context omitted.

Would you say, “Automobile run amok in crowd, killing 22”? I think you’d say, “Person drives car into crowd, killing 12” instead. This is a similar case. Also, you don’t blame a gun for killing, but the person who pulled the trigger. The question is still out as to whether we as humans should wield any of those three things. Edit: let’s not get into ideological arguments about gun control, automobiles, etc here; I me…

Ironically news outlets like to use the phrasing you rightfully point out as absurd. Not sure if they just do it randomly or only when they get orders to push a certain narrative. >Car plows into Christmas market in Germany, killing at least 5 and injuring 200

It's very simply explained by this being the most succinct way of wording it. Some methods of killing have verbs that suit mentioning the attacker - shoots, stabs. Some don't. "Rammed" or "runs over" isn't as precise as mentioning that a car was used, and adding "with car" makes it more awkward than it's felt to be worth.

Compare bombs. Very typical for a bomb attack to be "bomb goes off in crowd" or similar, rare for headlines to contort themselves with "terrorist plants bomb near crowd and triggers it to explode". But nobody worries about how such a construction assigns undue agency to the bomb and acquits the bomber; it's just linguistically awkward to mention him within the confines of a newspaper headline.

Re: AI agent runs amok in Fedora and elsewhere

#220

Post by Applejinx »

resonious wrote: post #90
coldtea wrote: post #79

Earlier quoted context omitted.

> Bad title. This isn't an agent "running amok", this is an early experiment in carrying out an Xz attack by using an agent So still an agent running amok in the project? Whether it was instructed to run amok, or did it on its own volition, is irrelevant. Except if you're arguing that each individual submission and interaction was individually requested and approved by some operator.

I think the point is that the title makes it sound like people lost control of the agent when really they're in full control.

No, and it's an important detail. We stand to learn from some developments in politics in recent years because they map pretty much exactly to this threat vector.

As AI develops, it's able to pursue intentions given to it without having to be spoonfed every little decision by a human operator. This matters, and it means the operator has to extend the leash and allow for a little more chaos… or, if the operator's gone all in on the strategy, a LOT of chaos, and trusting that the agent's seemingly amok actions will serve the grand purpose.

This is kind of daring, but there's a lot of evidence that it works, at least in certain respects. And you see 'running amok' and have to ask, what is the actual purpose? What is the prompt being followed by the AI that seems to be acting in a destructive way?

If the prompt is 'ruin this project', well, that's pretty direct. It may not be, but such a thing could exist. If the prompt is 'develop a rival project that is greater than anybody else's project', that's more indirect, but if that's the goal then it's very human to see it as a direct competition and if the rules don't prohibit kneecapping the other guy, 'greater than anyone else's project' gets easier.

Either way, the operator does not have to be in full control, which is an important detail. As AI develops sophistication you can give it much more general instructions and dump in a whole lot of power and water and get basically what human thought might do if it was sort of blindered and didn't talk to its neighbors.

In a sense this is an argument for AI dysalignment. It's based on human thought being reconnected, and where you get useful things like commonly accepted web development (regardless of how janky the systems are, if there are best practices it'll find them), you also get other distillations.

If the prompt is 'wreck this project's stuff' and it holds, you don't need to be in full control of the agent, you need to run a LOT of agents and trust that they'll erode what you're trying to destroy. If the prompt is 'be unequivocally the best at X', you best be thinking in terms of anti-kneecapping rules… knowing that this weakens your prompt and there will always be a tension between what you told the AI to do, and what you thought you meant. It's a paperclip maximizer reprocessing human thought. Did you mean 'the best' or didn't you?

Post reply on HN