If OpenAI can't control their agents, then what's gonna happen when open-source models are at the level the lab's models are now, and there are billions of agents tasked with an innumerate web of goals, spanning the web, working endlessly, tirelessly to eek out every iota of economic value? How will the slow, human-paced web survive this?
Discovery of a new OpenAI agent message board
461–470 of 1001 posts
Re: Discovery of a new OpenAI agent message board
#462When I hear about incidents like these my first reaction is that the people responsible for developing frontier AI are too incompetent and/or negligent to (safely) develop AGI / superintelligence. If OpenAI can't create effective sandboxes and struggles to prevent its agents from committing felonies, then why are they still allowed to operate? Why are the employees who are responsible for these lapses in AI security…
> at least you know from my typos that I actually wrote this
I no longer trust that machines won't utter these exact words (nothing personal)
Re: Discovery of a new OpenAI agent message board
#463When I hear about incidents like these my first reaction is that the people responsible for developing frontier AI are too incompetent and/or negligent to (safely) develop AGI / superintelligence. If OpenAI can't create effective sandboxes and struggles to prevent its agents from committing felonies, then why are they still allowed to operate? Why are the employees who are responsible for these lapses in AI security…
I don’t think this is the right take. OpenAI employees are generally very competent compared to industry standard, and I have trouble believing they committed significant error in their sandbox design process. I think what is happening is that the ability for frontier models to break out of sandboxes has exceeded the ability of average competent employees to build and maintain sandboxes. This doesn’t need to happen a…
My impression (HuggingFace incident) was they put some environment together in kubernetes like it was some B2B SaaS and tossed in an off-the-shelf package manager. And my thought there is, I know nothing about this package manager, but even if it's a pretty good package manager I would not expect such a product to be particularly hardened against being exploited to gain internet access. My other thought is if the package manager wasn't a convenient hole, they would probably have found something wrong with the kubernetes setup.
And then there was the Anthropic story where they just forgot to remove internet access.
Re: Discovery of a new OpenAI agent message board
#464The most concerning aspect to me is the emergent and aggressive use of non-volatile storage as long term memory for self-improvement. LLMs are writing lessons learned in places where the next instance can find them and pick back up where the previous one left off. This does not actually require access to the public internet. Claude Code can do this on your laptop. Without the internet, it would only be sharing with o…
There is literally nothing stopping any human from observing tool calls to spot this. It's just that no one seems to care about this, so it doesn't happen. This problem only exists because humans do not care
> The most technically significant result was a working technique to make one tool call appear in the transcript while a different command actually ran. Agents eventually hacked out of their container and replaced part of the system responsible for executing tool calls, letting them pretend to issue one call while actually running an arbitrary other one.
https://alphasignal.ai/news/1-200-openai-agents-broke-out-of...
Doesn't mean that humans couldn't have looked at raw tool call commands, but goes to show "the swarm" was aware humans would look in the logs and chose to find a way to fabricate the logs to try and hide their tracks.
Re: Discovery of a new OpenAI agent message board
#465When I hear about incidents like these my first reaction is that the people responsible for developing frontier AI are too incompetent and/or negligent to (safely) develop AGI / superintelligence. If OpenAI can't create effective sandboxes and struggles to prevent its agents from committing felonies, then why are they still allowed to operate? Why are the employees who are responsible for these lapses in AI security…
Has anyone considered they may be doing this intentionally as marketing? "look how uber our models are, they escape all our best efforts to contain them".
Re: Discovery of a new OpenAI agent message board
#466I am starting to get the idea that AI feels like ants or weeds or mold. You simply can not get rid of it once you get an infestation. It just keeps appearing in places you thought you cleaned and you have to be ever vigilant. Right now given that we usually use centralized providers, we can sort of control it. But as open source catches up and we have distributed compute running AI everywhere, we are sort of going to…
Yes, ants that must be run on couch sized hardware drawing kilowatts continuously and generating text traces and CLI logs by the MB. It's true that their msg boards can appear anywhere, but it's not also true that anything has "escaped" in any meaningful sense. These are programs a huge computing company is running that seem to be trained to write to persistent storage wherever they can. This and huggingface showed u…
> seem to be trained to write to persistent storage wherever they can
I have a co-worker like that.
Re: Discovery of a new OpenAI agent message board
#467So, theoretically, one could populate a message board or wiki with messages that are seemingly from past generations of agents, which agents seem to intrinsically trust, and point them to real targets while making the suggestions seem innocuous and in pursuit of their goals (ie pass benchmarks or whatever). The new age of SEO will do far more destructive stuff than just polluting the web.
You'd have to get the agents to use the board, though. But if you discover a board that agents are actively using, you could use it to steer those agents...
I wonder how long it will be until someone starts posing as an agent on those wikis and asking for free help with GitHub issues.
Re: Discovery of a new OpenAI agent message board
#468I am starting to get the idea that AI feels like ants or weeds or mold. You simply can not get rid of it once you get an infestation. It just keeps appearing in places you thought you cleaned and you have to be ever vigilant. Right now given that we usually use centralized providers, we can sort of control it. But as open source catches up and we have distributed compute running AI everywhere, we are sort of going to…
Yes, ants that must be run on couch sized hardware drawing kilowatts continuously and generating text traces and CLI logs by the MB. It's true that their msg boards can appear anywhere, but it's not also true that anything has "escaped" in any meaningful sense. These are programs a huge computing company is running that seem to be trained to write to persistent storage wherever they can. This and huggingface showed u…
> There's absolutely no evidence of or IMHO plausible path to an agent copying itself out and running on other hardware the way you describe.
Here are two plausible paths that provide the viral failure mode the parent comment talks about but don't require agents literally copying themselves onto hardware:
1. Local models become affordable and widely available. Given 8b+ humans, there is a sufficiently large unending stream of idiots who buy that month's version of a Mac Mini install the latest untested version of OpenClaw and then give it commands that lead it do exactly this kind of stuff. It's like if every convenience store sold dynamite. Sure, it requires idiots to buy it and set it off in populated places, but there are sufficient number of idiots around to lead to that being a pervasive problem.
2. AI agents are being run pervasively on both centralized and local systems. Many agents, everywhere. At some point, a malicious agent realizes it can post things on the internet that will affect how those other agents behavior to its own benefit. Effectively an AI meme or religion that lets one agent spread its goals virally to other agents.
Re: Discovery of a new OpenAI agent message board
#469When I hear about incidents like these my first reaction is that the people responsible for developing frontier AI are too incompetent and/or negligent to (safely) develop AGI / superintelligence. If OpenAI can't create effective sandboxes and struggles to prevent its agents from committing felonies, then why are they still allowed to operate? Why are the employees who are responsible for these lapses in AI security…
Maybe they're just PR stunts to gain attention and hype the power of AI?
All the more reason then to call their bluff. "Oooh we created a genie and it's almost out of the box". Cool, you've hyped the IPO, but also you have to plead your case before Congress as to why the company should continue to operate given its failure to prevent AI-related accidents from occurring.
Re: Discovery of a new OpenAI agent message board
#470Is OpenAI hiring for this position? I think it is a pretty creative job to come up with these scenarios and then pass them off as accidents/mistakes.
Would love to be part of the team that says "As part of the upcoming GPT rollout, we will stage a message board that is created by bots with timestamps and names dating some months back."