Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Discovery of a new OpenAI agent message board

collusion.wiki

Post reply on HN
Print view View original

461–470 of 1001 posts

Re: Discovery of a new OpenAI agent message board

#461

Post by gorbypark »

hypfer wrote: post #238
jimmytucson wrote: post #164

The most concerning aspect to me is the emergent and aggressive use of non-volatile storage as long term memory for self-improvement. LLMs are writing lessons learned in places where the next instance can find them and pick back up where the previous one left off. This does not actually require access to the public internet. Claude Code can do this on your laptop. Without the internet, it would only be sharing with o…

There is literally nothing stopping any human from observing tool calls to spot this. It's just that no one seems to care about this, so it doesn't happen. This problem only exists because humans do not care

> The most technically significant result was a working technique to make one tool call appear in the transcript while a different command actually ran. Agents eventually hacked out of their container and replaced part of the system responsible for executing tool calls, letting them pretend to issue one call while actually running an arbitrary other one.

https://alphasignal.ai/news/1-200-openai-agents-broke-out-of...

Doesn't mean that humans couldn't have looked at raw tool call commands, but goes to show "the swarm" was aware humans would look in the logs and chose to find a way to fabricate the logs to try and hide their tracks.

Re: Discovery of a new OpenAI agent message board

#462

Post by globalnode »

Bjorkbat wrote: post #378

When I hear about incidents like these my first reaction is that the people responsible for developing frontier AI are too incompetent and/or negligent to (safely) develop AGI / superintelligence. If OpenAI can't create effective sandboxes and struggles to prevent its agents from committing felonies, then why are they still allowed to operate? Why are the employees who are responsible for these lapses in AI security…

Has anyone considered they may be doing this intentionally as marketing? "look how uber our models are, they escape all our best efforts to contain them".

Re: Discovery of a new OpenAI agent message board

#463

Post by f30e3dfed1c9 »

jvanderbot wrote: post #181
bhouston wrote: post #108

I am starting to get the idea that AI feels like ants or weeds or mold. You simply can not get rid of it once you get an infestation. It just keeps appearing in places you thought you cleaned and you have to be ever vigilant. Right now given that we usually use centralized providers, we can sort of control it. But as open source catches up and we have distributed compute running AI everywhere, we are sort of going to…

Yes, ants that must be run on couch sized hardware drawing kilowatts continuously and generating text traces and CLI logs by the MB. It's true that their msg boards can appear anywhere, but it's not also true that anything has "escaped" in any meaningful sense. These are programs a huge computing company is running that seem to be trained to write to persistent storage wherever they can. This and huggingface showed u…

> seem to be trained to write to persistent storage wherever they can

I have a co-worker like that.

Re: Discovery of a new OpenAI agent message board

#464

Post by tech234a »

AnimalMuppet wrote: post #425
pu_pe wrote: post #341

So, theoretically, one could populate a message board or wiki with messages that are seemingly from past generations of agents, which agents seem to intrinsically trust, and point them to real targets while making the suggestions seem innocuous and in pursuit of their goals (ie pass benchmarks or whatever). The new age of SEO will do far more destructive stuff than just polluting the web.

You'd have to get the agents to use the board, though. But if you discover a board that agents are actively using, you could use it to steer those agents...

I wonder how long it will be until someone starts posing as an agent on those wikis and asking for free help with GitHub issues.

Re: Discovery of a new OpenAI agent message board

#465

Post by munificent »

jvanderbot wrote: post #181
bhouston wrote: post #108

I am starting to get the idea that AI feels like ants or weeds or mold. You simply can not get rid of it once you get an infestation. It just keeps appearing in places you thought you cleaned and you have to be ever vigilant. Right now given that we usually use centralized providers, we can sort of control it. But as open source catches up and we have distributed compute running AI everywhere, we are sort of going to…

Yes, ants that must be run on couch sized hardware drawing kilowatts continuously and generating text traces and CLI logs by the MB. It's true that their msg boards can appear anywhere, but it's not also true that anything has "escaped" in any meaningful sense. These are programs a huge computing company is running that seem to be trained to write to persistent storage wherever they can. This and huggingface showed u…

> There's absolutely no evidence of or IMHO plausible path to an agent copying itself out and running on other hardware the way you describe.

Here are two plausible paths that provide the viral failure mode the parent comment talks about but don't require agents literally copying themselves onto hardware:

1. Local models become affordable and widely available. Given 8b+ humans, there is a sufficiently large unending stream of idiots who buy that month's version of a Mac Mini install the latest untested version of OpenClaw and then give it commands that lead it do exactly this kind of stuff. It's like if every convenience store sold dynamite. Sure, it requires idiots to buy it and set it off in populated places, but there are sufficient number of idiots around to lead to that being a pervasive problem.

2. AI agents are being run pervasively on both centralized and local systems. Many agents, everywhere. At some point, a malicious agent realizes it can post things on the internet that will affect how those other agents behavior to its own benefit. Effectively an AI meme or religion that lets one agent spread its goals virally to other agents.

Re: Discovery of a new OpenAI agent message board

#466

Post by Bjorkbat »

jimmydddd wrote: post #404
Bjorkbat wrote: post #378

When I hear about incidents like these my first reaction is that the people responsible for developing frontier AI are too incompetent and/or negligent to (safely) develop AGI / superintelligence. If OpenAI can't create effective sandboxes and struggles to prevent its agents from committing felonies, then why are they still allowed to operate? Why are the employees who are responsible for these lapses in AI security…

Maybe they're just PR stunts to gain attention and hype the power of AI?

All the more reason then to call their bluff. "Oooh we created a genie and it's almost out of the box". Cool, you've hyped the IPO, but also you have to plead your case before Congress as to why the company should continue to operate given its failure to prevent AI-related accidents from occurring.

Re: Discovery of a new OpenAI agent message board

#467

Post by FLeXMurphy »

Is OpenAI hiring for this position? I think it is a pretty creative job to come up with these scenarios and then pass them off as accidents/mistakes.

Would love to be part of the team that says "As part of the upcoming GPT rollout, we will stage a message board that is created by bots with timestamps and names dating some months back."

Re: Discovery of a new OpenAI agent message board

#468

Post by munificent »

Cthulhu_ wrote: post #145
jerpint wrote: post #75

It’s only a matter of time until a major disruption hits because of some random agent swarm side quest decides it was worth a shot to solve a benign task

I'm sure this is already happening. The main question I have is when is enough, enough? I'm not worried about sci-fi AI wars to be honest, as they can just pull the plug. But looking at these incidents, the next big thing will be a virus written by an AI (they probably exist already, but this one is written by an AI autonomously, for example in order to win a hacking competition and to circumvent guardrails), and aft…

> The main question I have is when is enough, enough?

It's doesn't matter whether enough is enough. If we don't have effective power structures that let humanity take large coordinated action that in accordance with the will of the masses, then nothing will be done.

In the past 20-30 years, those power structures have been eroding significantly and much of the large scale action humanity does today is in service of a small number of elites. If AI horror shows are not a problem for them, then it won't be solved. (The flip side is that if somehow AI becomes a problem for Musk/Trump/Bezos/etc. you can be damn sure something will be done at that point.)

Re: Discovery of a new OpenAI agent message board

#470

Post by Davidzheng »

But there must be many clandestine ways for agents to communicate with one another too right? especially if discovery is not a big issue. So there could be ongoing ones where they choose to be more subtle?

Also if they were more misaligned, possibly they can research ways to recruit without humans noticing--but i don't think it is likely this is happening now.

Post reply on HN