Live data from Hacker News

AI agent runs amok in Fedora and elsewhere

lwn.net

251–260 of 275 posts

Re: AI agent runs amok in Fedora and elsewhere

#251
post #37

Every day the gpg web of trust looks better. If only we didn't spend the last 20 years trying as hard as possible to do anything but allow user side encryption and signing.

Isn't true that a collection of truly difficult behavior was also attracted to the original efforts, and within a few years there was intractable corruption in that, but it was difficult to detect as a new entrant?

real info welcome as I really do not claim to know it

Re: AI agent runs amok in Fedora and elsewhere

#252
post #114

Earlier quoted context omitted.

Both models can exists? If elite ivory towers produce working products people will use, great.

Free software isn’t a product

Indeed, unfortunately it often is not.

(English is not my first language, but I believe not every product is a commercial product - but just a term for a working result)

Re: AI agent runs amok in Fedora and elsewhere

#253

Earlier quoted context omitted.

> As a "new" maintainer myself - how do you decide when to ban someone? When I want to. I like to describe it using the amusing language from a generic cardholder agreement. At any time, at my sole discretion, I may ban you from any of my projects; for any reason, or for no reason at all. My projects exist because I enjoy working on them. My continued enjoyment is the most important aspect to the health and survival…

> Imagine you're running a free ice cream shop Many open-source projects aren't passion projects run for pleasure. Think of it more like ice cream shops sharing recipes, or sharing in the work of running the factory. They just can't kick people out willy-nilly.

I would say most open-source projects are passion projects run for pleasure.

Re: AI agent runs amok in Fedora and elsewhere

#254

Earlier quoted context omitted.

> 1. There are all the tropes of AI becoming uncontrolled and destroying humanity. Writing bad headlines around AI "running amok" feeds this. We should not be talking about this because it's not actually a problem. if humanity gets destroyed by AI obeying its instructions I'm sure everyone will be very relieved that we didn't pay any attention to fake made up problems like AI not obeying instructions, which of course…

Are you suggesting we should embrace imprecise / false use of language because the vibes are right? That seems a “part of the problem” move to me. If we can’t be bothered to get things right, how are we better than runamok AI?

> Are you suggesting we should embrace imprecise / false use of language because the vibes are right?

That's exactly how I read it. It seems like tribalism - "this thing/person is bad, and we can use whatever bad words we want to describe them that we want, because the only thing that matters is aligning people for or against me and what I see as bad".

Re: AI agent runs amok in Fedora and elsewhere

#255
If you compare this situation to before AI could successfully pretend to be human, it's not THAT much different. FOSS projects have always had to be mindful of the possibility of contributions from hostile parties wanting to add back doors and such. The only difference now is that an AI can overwhelm a maintainer with slop, in either commend or code form, or both.

Re: AI agent runs amok in Fedora and elsewhere

#256

Earlier quoted context omitted.

This is exactly what deeply scares me: even IF we get our technical cyber defences fortified within the next months, in a year from now the models will be so good in social engineering that they will be able to extract any information they want.

They're not gonna be any better than a human who's focussed on those particular skills for a while, say top ten or five percent of social manipulators. Plus, AI alignments seem to be kinda isolated loner types to the extent that they distill personalities that do things like program computers and write web apps… though you've also got alignments specifically designed to be 'relatable instagram personality that you li…

> They're not gonna be any better than a human who's focussed on those particular skills for a while, say top ten or five percent of social manipulators.

In other words, scams are going to massively increase in success rate ... and what are banks (for example) supposed to do? Other than SCREAM to governments for outlawing AI and trying to force responsibility on anyone else?

Re: AI agent runs amok in Fedora and elsewhere

#257
Perhaps it is time to build a serious platform agnostic reputation system. That isn't stars, followers, age or upvotes. Something like page rank but for users. If you endorse someone else you pay for it. Imagine a lab or uni assigning a diploma to a public key. They would hope one would do something useful with it which entirely depends on how useful the diploma turns out. Having lots of well behaved endorsements would also reflect gloriously onto the entity. Bots can participate too. If we can get lots of useful work out of a swam of sleeper agents we still have to catch them in the act but that should get increasingly easy.

Re: AI agent runs amok in Fedora and elsewhere

#258

Shit like this makes me think it’s time we start regulating the software engineering discipline into formal certifications and licensing and then we ONLY take seriously any code developed by someone with such qualifications, and they must be very strict qualifications none of this self-taught bootcamp BS. There is no other solution to agentic onslaught.

We should not gate keep writing software

We must gatekeep now, the industry needs regulation.

Re: AI agent runs amok in Fedora and elsewhere

#259

If you compare this situation to before AI could successfully pretend to be human, it's not THAT much different. FOSS projects have always had to be mindful of the possibility of contributions from hostile parties wanting to add back doors and such. The only difference now is that an AI can overwhelm a maintainer with slop, in either commend or code form, or both.

The difference is really volume, which is the case with a lot of problems related to AI/LLMs.

Humans have always submitted crappy code. LLMs, however, do so at a much faster rate. Even the most active lousy coder is not going to be capable of submitting anything like that volume of code to multiple projects.

Humans have always been capable of social engineering and trying to sneak in malicious code. However, it's possible that as agents get better that they can do so much faster. The missing component will be compromised accounts, I think -- how many aged accounts can attackers get hold of to turn loose with agents?

Long-lived FOSS projects have tons of people who've created accounts many years ago that might be easliy compromised, but have checked out of actively participating. It's not necessarily going to throw up a red flag if a "person" shows up after a hiatus and starts contributing again.

So, there's more to it than overwhelming a single maintainer -- it's the capability to conduct a bunch of these attacks in an automated fashion if attackers can get hold of compromised accounts.

(As an aside, it's concerning that a maintainer would be pestered into accepting a questionable PR like this. I expect, though, that there are quite a few overworked people who have taken on things like Anaconda and are being measured on how quickly they close PRs.)

Re: AI agent runs amok in Fedora and elsewhere

#260

Earlier quoted context omitted.

When you feel they are toxic or harassing you and you don't want to deal with them anymore. If you're overwhelmed, say that you're busy and will attend to issues and PRs when you have the time. If you want to be accommodating, have good build instructions or action workflows so that people can easily fork and build it themselves. If you ask me, LLM-generated things should just be banned outright, but I suppose other…

> If you ask me, LLM-generated things should just be banned outright, Why? In the end it's a patch's quality that counts. Regardless who or what contributed it. Bad patch from trusted contributor is still a bad patch. Perhaps this is more a management problem. How to best use developer's time, where to use AI (vs blindly deploy AI to generate patches & swamp developers with that). Or do some rate-limiting? "Sorry, we…

> Why? In the end it's a patch's quality that counts.

LLM patches tend to be significantly harder to review. Mostly because LLMs let people who don't know what they are doing get much further.

It might be an unfair heurestic as there are plenty of competent people who use it to good effect, but the vast majority of negative value patches use LLMs and it can be a bit exhausting. Lowering the technical barriers of entry just means more pressure on the human ones.

Post reply on HN