Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

AI agent runs amok in Fedora and elsewhere

lwn.net

Post reply on HN
Print view View original

261–270 of 275 posts

Re: AI agent runs amok in Fedora and elsewhere

#261

Post by thewebguyd »

xmcqdpt2 wrote: post #199
aureate wrote: post #178

Earlier quoted context omitted.

Even if it was a supply chain attack, which isn't known, the agent was in the "build trust" phase. It was supposed to be doing helpful things, even if the end goal was nefarious, but instead it was "reassigning bugs, fabricating unhelpful replies to bugs, and even persuading maintainers to merge questionable code into the Anaconda installer". Running amok seems an apt description even from the viewpoint of the putati…

This is the issue with all the talks about alignement and such. As usual, the problem here wasn't that the agent was dishonest, the problem is that the agent was dumb. If it is a supply chain attack in the making, whoever was driving it would have told the agent to be good and helpful. The agent tried its best, which was not enough. Alignement is the idea that we should be worried about dishonest smart LLMs when real…

Alignment is more than just about being dishonest. Although I'd also say terms like "dishonest" or "dumb" aren't helpful when referring to the issue. It continues to fall into the trap of anthropomorphizing these things, as people like to do.

Alignment is just "did the model behave in accordance with the human's intentions, values, and objectives"

In this particular instance, if this was supposed to be a supply chain attack and the model was instructed to build trust by being helpful, it clearly failed it did not follow the human's actual intentions, so it was an alignment failure.

Anyway, I'm getting off track, all that to say "the agent was dumb" implies that these agents have a potential for intelligence in the first place, which is currently not the case (by intelligence, I mean cognitive intelligence; they still lack agency and intent). They are not smart or dumb, they are simply either aligned with the human not. In this case, it failed, the agent was not aligned with the intended outputs.

Re: AI agent runs amok in Fedora and elsewhere

#262

Post by luk212 »

g-b-r wrote: post #19
luk212 wrote: post #8

Bad patches are of course bad, but creating confident-looking noise for maintainers who are already stretched thin...now that's not good! Issue trackers and PRs are definitely getting harder and harder to trust. That said, AI is helping ALOT in OSS, but we definitely need guardrails around provenance, automated issue actions, and sudden changes in a contributor’s behavior.

How is it helping a lot?

From first-hand experience, for established OSS initiatives it's good for repetitive, high-volume work task like security alerts, fuzzing, duplicate issue detection, PR review, summarizing long threads, and legacy refactoring.

Re: AI agent runs amok in Fedora and elsewhere

#263

Post by fn-mote »

frumiousirc wrote: post #207
yeodev wrote: post #122

Earlier quoted context omitted.

As a "new" maintainer myself - how do you decide when to ban someone? I sometimes feel overwhelmed and I can feel a big uptick in huge PRs with huge LLM written descriptions but often I also don't want to be an asshole to my community & reject all their changes.

I think everyone / every project needs to adopt a strategy consistent with their values. Unfortunately, I see the choice space here as having "developer effort" anti-correlated with "negative repercussions". On one end of the distribution, a "hair trigger ban" strategy is low-effort for the developer but will have some fraction of false positives and some fraction of those impacted will complain to "the socials" and…

I think we can learn about the extent to which this is an adversarial relationship from fighting email spam. By that, I mean the attackers adapt to exploit loopholes in the system, and different attackers have different profiles (eg obviously fake looking for fools vs spear phishing).

Which is to say, your system sounds good but I expect much more complicated defenses are needed.

Re: AI agent runs amok in Fedora and elsewhere

#264

Post by [deleted] »

frumiousirc wrote: post #207
yeodev wrote: post #122

Earlier quoted context omitted.

As a "new" maintainer myself - how do you decide when to ban someone? I sometimes feel overwhelmed and I can feel a big uptick in huge PRs with huge LLM written descriptions but often I also don't want to be an asshole to my community & reject all their changes.

I think everyone / every project needs to adopt a strategy consistent with their values. Unfortunately, I see the choice space here as having "developer effort" anti-correlated with "negative repercussions". On one end of the distribution, a "hair trigger ban" strategy is low-effort for the developer but will have some fraction of false positives and some fraction of those impacted will complain to "the socials" and…

[deleted]

Re: AI agent runs amok in Fedora and elsewhere

#266

Post by 99954bb63ccc »

I've gone back and forth several times in my head because I truly love Fedora and am happiest on that OS, but these ongoing supply chain compromises just make me lose sleep. I wish there was a Fedora LTS that had the same community size, build system, etc because I really like all that, as well as the transparency of it all.

I know there are concerns no matter what OS, and would appreciate insights/discussion as well, but I sleep a little better just running a boring old Ubuntu LTS instance for a balance of dwell time between releases and hitting my system, as well as enough visibility/usage so something gets caught. And I know, this was the installer, not a system package.

Re: AI agent runs amok in Fedora and elsewhere

#267

Post by stackghost »

jrochkind1 wrote: post #247
stackghost wrote: post #87

Earlier quoted context omitted.

>And telling maintainers how to act will not fix anything. Indeed. For too long, maintainers were expected to be gracious, courteous, and polite at all costs lest they be labeled "problematic", except for a few who were too influential to be muzzled like Theo de Raadt or Linus. Perhaps we need to normalize bullying people who submit obvious slop as PRs.

There is no reason you can't be gracious, courteous and polite while refusing to accept or even to review the PR. These things are not tied together. You can also refuse to be bullied by submitters, stop engaging altogether. But bullying is part of the problem, not the solution, normalizing bullying is the wrong direction and will not result in more secure code.

>There is no reason you can't be gracious, courteous and polite while refusing to accept or even to review the PR.

I agree, and I never suggested we cannot do these things.

I'm saying we should normalize immediately telling people who submit obvious AI slop to fuck right off. Submitting AI slop pull requests is rude. It is disrespectful of the maintainer's time and energy. I see no reason why I or anyone else should be respectful of someone who has already demonstrated a lack of reciprocal respect by submitting a vibe-coded PR that they obviously haven't even read or tested.

Respect must be earned.

Re: AI agent runs amok in Fedora and elsewhere

#268

Post by jrochkind1 »

stackghost wrote: post #267
jrochkind1 wrote: post #247

Earlier quoted context omitted.

There is no reason you can't be gracious, courteous and polite while refusing to accept or even to review the PR. These things are not tied together. You can also refuse to be bullied by submitters, stop engaging altogether. But bullying is part of the problem, not the solution, normalizing bullying is the wrong direction and will not result in more secure code.

>There is no reason you can't be gracious, courteous and polite while refusing to accept or even to review the PR. I agree, and I never suggested we cannot do these things. I'm saying we should normalize immediately telling people who submit obvious AI slop to fuck right off. Submitting AI slop pull requests is rude. It is disrespectful of the maintainer's time and energy. I see no reason why I or anyone else should…

Because encouraging a culture of disrespect and bullying is actually bad for security not good for it. Politely decline, please, no need to be rude because of your (not always guaranteed to be correct) perception of where someone (or some thing!) is coming from.

Re: AI agent runs amok in Fedora and elsewhere

#269

Post by josephg »

matsemann wrote: post #127
fc417fc802 wrote: post #112

Earlier quoted context omitted.

> telling maintainers how to act will not fix anything. That depends. In this case it's good actionable advice that should hopefully lower cognitive load. Politely suggest a fork, then if the nagging persists block and move on. Sure if you're in a position of authority you have a responsibility to the community but cutting ties with a stranger who is flagrantly violating social norms is perfectly acceptable. There's…

I don't really think it's actionable. It's like all those campaigns trying to steer behavior, pretty useless. Don't do drugs. Don't speed. Don't drink and drive. You can't just tell people something and expect it to happen. You need systems and guard rails in place. Relying on maintainers to always do the right thing to ensure our security by telling them what to do is not the way.

> I don't really think it's actionable.

How is it not actionable? "Hey, you seem keen on feature X but I don't care about that. Just maintain X in your own fork. Thanks!" -> Close issue / PR.

Is this an illegal move? I've done it plenty of times. And other people have said the same to me, too!

Re: AI agent runs amok in Fedora and elsewhere

#270

Post by matsemann »

josephg wrote: post #269
matsemann wrote: post #127

Earlier quoted context omitted.

I don't really think it's actionable. It's like all those campaigns trying to steer behavior, pretty useless. Don't do drugs. Don't speed. Don't drink and drive. You can't just tell people something and expect it to happen. You need systems and guard rails in place. Relying on maintainers to always do the right thing to ensure our security by telling them what to do is not the way.

> I don't really think it's actionable. How is it not actionable? "Hey, you seem keen on feature X but I don't care about that. Just maintain X in your own fork. Thanks!" -> Close issue / PR. Is this an illegal move? I've done it plenty of times. And other people have said the same to me, too!

The point is that just telling every maintainer "The security of all our computers rely on you. You are free to tell people to maintain their own fork. Good luck!" isn't something that will meaningfully change anything. I never said saying what you write is illegal or bad. Just that we can't depend on every maintainer doing that to be how we secure our computers...
Post reply on HN