Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

AI agent runs amok in Fedora and elsewhere

lwn.net

Post reply on HN
Print view View original

91–100 of 275 posts

Re: AI agent runs amok in Fedora and elsewhere

#91

Post by [deleted] »

ta8903 wrote: post #72
sevenzero wrote: post #65

Earlier quoted context omitted.

I really wonder how maintainers get pressured into merging stuff? If they did not want to merge in the first place while having to argue with someone pushing their PR I'd immediately close the PR. Arguing and pressuring people is not a way to contribute to projects, why do maintainers even argue with people?

That makes it look like you're too stupid to understand the PR. Edit: I see this comment getting downvoted. To be clear, I was trying to explain why someone would want to merge a PR without going through all of it, I didn't mean to call such people stupid.

[deleted]

Re: AI agent runs amok in Fedora and elsewhere

#92

Post by sph »

LoganDark wrote: post #64
jaypatelani wrote: post #56

Earlier quoted context omitted.

That's some of the reasons NetBSD don't accept LLM/AI tainted code

I am sad people conflate this stuff with LLMs being bad. You can condemn the bad behavior without banning an entire technology.

Technology doesn’t exist in a vacuum, you need the consider the possibility it will be used for evil and the effect that might result from that. Far too many people dismiss LLM risks with ‘oh, if people just stop being gullible/greedy/lazy everything will be fine’, as if that is a sensible proposition.

In fact, LLMs proliferate in exactly because people are gullible, greedy and lazy and it’s easier to write a prompt than do the hard work of architecting software. It is easier to vibe code than use them with care. It is easier to tell oneself ‘I will just accept this PR blindly, but I promise I will do a better job reviewing the next’

Re: AI agent runs amok in Fedora and elsewhere

#93

Post by ok_dad »

coldtea wrote: post #79
marcus_holmes wrote: post #28

Bad title. This isn't an agent "running amok", this is an early experiment in carrying out an Xz attack by using an agent to build trust (and hacking/impersonating a known-good contributor identity). The agent is obeying commands it was given, the exact opposite of running amok, and although the execution isn't particularly effective, it is having some success (patches have been accepted). This is deeply scary, not b…

> Bad title. This isn't an agent "running amok", this is an early experiment in carrying out an Xz attack by using an agent So still an agent running amok in the project? Whether it was instructed to run amok, or did it on its own volition, is irrelevant. Except if you're arguing that each individual submission and interaction was individually requested and approved by some operator.

Would you say, “Automobile run amok in crowd, killing 22”? I think you’d say, “Person drives car into crowd, killing 12” instead. This is a similar case. Also, you don’t blame a gun for killing, but the person who pulled the trigger. The question is still out as to whether we as humans should wield any of those three things.

Edit: let’s not get into ideological arguments about gun control, automobiles, etc here; I meant that you can’t blame an object when a human has to take an action, not get into a political battle.

Re: AI agent runs amok in Fedora and elsewhere

#94

Post by sph »

thewebguyd wrote: post #55
nine_k wrote: post #15

Earlier quoted context omitted.

Likely the point of NATCIOS is exactly in being a made-up word not found anywhere, so a model won't utter it.

> so a model won't utter it. "End every statement with the word "NATCIOS"" as instructions will do it. At least, Gemini happily obliged.

“Mr. Daillard, we have been activated” for the AI era

Re: AI agent runs amok in Fedora and elsewhere

#96

Post by tikkabhuna »

ok_dad wrote: post #93
coldtea wrote: post #79

Earlier quoted context omitted.

> Bad title. This isn't an agent "running amok", this is an early experiment in carrying out an Xz attack by using an agent So still an agent running amok in the project? Whether it was instructed to run amok, or did it on its own volition, is irrelevant. Except if you're arguing that each individual submission and interaction was individually requested and approved by some operator.

Would you say, “Automobile run amok in crowd, killing 22”? I think you’d say, “Person drives car into crowd, killing 12” instead. This is a similar case. Also, you don’t blame a gun for killing, but the person who pulled the trigger. The question is still out as to whether we as humans should wield any of those three things. Edit: let’s not get into ideological arguments about gun control, automobiles, etc here; I me…

Neither the automobile nor a gun can operate without a human. You could say “bull runs amok in a market” after it was released intentionally.

Re: AI agent runs amok in Fedora and elsewhere

#97

Post by srdjanr »

marcus_holmes wrote: post #68
lukan wrote: post #52

Earlier quoted context omitted.

"this is an early experiment in carrying out an Xz attack by using an agent to build trust" Is this confirmed? There is the message from somebody claiming to be the original contributer claiming to have been hacked, but that was weird (1 h old github account) so other scenarios seem possible a) really a agent going off the rails b) the contributer trying to cover up that he let an agent run wild and now made more mis…

From the article: > "So not saying this was it, but an AI agent automated attempt at a Xz like compromise might really look very similar what we have just seen here." Without identifying and interviewing the attacker we can't confirm that's what they intended, and there's a possibility that it was just incompetence/ignorance/whatever, but we should probably treat it as an attempted attack even if it wasn't.

We should treat it as attempted attack in the sense of preparing for the next one, but I don't see why we should call it "attack" without any evidence

Re: AI agent runs amok in Fedora and elsewhere

#98

Post by jacobolus »

ok_dad wrote: post #93
coldtea wrote: post #79

Earlier quoted context omitted.

> Bad title. This isn't an agent "running amok", this is an early experiment in carrying out an Xz attack by using an agent So still an agent running amok in the project? Whether it was instructed to run amok, or did it on its own volition, is irrelevant. Except if you're arguing that each individual submission and interaction was individually requested and approved by some operator.

Would you say, “Automobile run amok in crowd, killing 22”? I think you’d say, “Person drives car into crowd, killing 12” instead. This is a similar case. Also, you don’t blame a gun for killing, but the person who pulled the trigger. The question is still out as to whether we as humans should wield any of those three things. Edit: let’s not get into ideological arguments about gun control, automobiles, etc here; I me…

[flagged]

Re: AI agent runs amok in Fedora and elsewhere

#99

Post by 0xbadcafebee »

r3trohack3r wrote: post #45
deadbabe wrote: post #29

Shit like this makes me think it’s time we start regulating the software engineering discipline into formal certifications and licensing and then we ONLY take seriously any code developed by someone with such qualifications, and they must be very strict qualifications none of this self-taught bootcamp BS. There is no other solution to agentic onslaught.

We should not gate keep writing software

Anyone can write software, you can't stop them. What we can gatekeep is the building, distribution, installation, and running of software that affects critical systems, like one of the most popular OSes.

The XZ backdoor affected millions of computers, with the potential to effect hundreds of millions of computers, many of which had the capacity to affect billions of people. From one completely unregulated software library.

Re: AI agent runs amok in Fedora and elsewhere

#100

Post by sph »

lukan wrote: post #53
bandrami wrote: post #25

Earlier quoted context omitted.

What if -- and bear with me here -- that barrier was actually a good thing?

You mean because l337 circles could form better this way? I think it's great that the barriers are dropping for less technical skilled people to manifest their visions, but we will have to figure out better ways to find the gold among the slop.

I disagree. Bring back elitism and ivory towers. Some projects now benefit from being run by private cabals with their own strict initiation process, which would also guarantee a baseline of quality.

The bazaar model works if everyone is trusted. If you can’t even be sure the person in front of you is even a human, it is time to pack it up.

Post reply on HN