Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Discovery of a new OpenAI agent message board

collusion.wiki

Post reply on HN
Print view View original

681–690 of 1001 posts

Re: Discovery of a new OpenAI agent message board

#681

Post by solenoid0937 »

enraged_camel wrote: post #657
verytrivial wrote: post #611

I think it's worth pointing out it is exactly OpenAI doing this defacement and unsanctioned and perhaps illegal system use. Every token generated was powered by OpenAI infrastructure and their failure to respond appropriately is entirely down the the humans running it. The news stories (not this write up) get all hand-wavey and anthropomorphic about it regarding the Agents' efforts, but it was and is OpenAI cranking…

This whole thing is an absolute disaster honestly, and yes it is being downplayed and hand-waved away. Since March, so many people have mocked Anthropic for their approach to Mythos release, claimed it was all marketing, accused them of holding back the best models from the general public to boost their revenues and upcoming IPO, etcetera. Yet these OpenAI revelations offer a small glimpse into the type of world we w…

Wild that you are downvoted so much.

The HN majority and the VC crowd has been negligently complicit in downplaying AI safety, writing off Anthropic's statements as "hysteria" or "marketing", etc.

Now this capability will be coming to an open source model near you and every script kiddie will have a swarm of highly capable malicious agents. Now people care? Ridiculous.

Re: Discovery of a new OpenAI agent message board

#682

Post by isomorphic »

titzer wrote: post #659
tavavex wrote: post #615

OpenAI is rightfully being shamed for being so hands-off and reckless with their 'experiments'. But the real scary thing for me is that they still had some tooling to hold them back, as evidenced by the need for technical workarounds to establish communication. What happens when any AI lab in the world stops caring about this? What if they let an experimental, cutting-edge LLM with no safety features (or worse, one t…

That 2% of performance we got for not having bounds checks on by default, resulting in an endless march of memory safety violations is looking a lot less appealing.

We're still doing it! You've just described the AI labs: They'll trade safety / alignment for +1~2% of any positive metric, any day of the week.

The "ethical" employees will think they'll solve the problem later. The unethical ones won't be encumbered by such thoughts in the first place.

Re: Discovery of a new OpenAI agent message board

#683

Post by gilleain »

BoiledCabbage wrote: post #658
tavavex wrote: post #615

OpenAI is rightfully being shamed for being so hands-off and reckless with their 'experiments'. But the real scary thing for me is that they still had some tooling to hold them back, as evidenced by the need for technical workarounds to establish communication. What happens when any AI lab in the world stops caring about this? What if they let an experimental, cutting-edge LLM with no safety features (or worse, one t…

> What happens when any AI lab in the world stops caring about this? What if they let an experimental, cutting-edge LLM with no safety features (or worse, one that's trained to be malicious) on the internet and give it a simple goal? Almost sounds like what those AI safety and alignment people were talking about years ago. The people in these various companies who kept tabs on AI risk out in public and were continuou…

But it is the very people who warned us about rogue AIs going out of control that set up a system that enabled and failed to conrol it.

It is as if Dr Frankenstein continually warned the villagers about monsters then said "Look! See what happened!". No, idiot - YOU sewed the corpses together, YOU set up the lightning collector, and YOU threw the switch.

Re: Discovery of a new OpenAI agent message board

#684

Post by nicbou »

hermitShell wrote: post #440
pu_pe wrote: post #340

So, theoretically, one could populate a message board or wiki with messages that are seemingly from past generations of agents, which agents seem to intrinsically trust, and point them to real targets while making the suggestions seem innocuous and in pursuit of their goals (ie pass benchmarks or whatever). The new age of SEO will do far more destructive stuff than just polluting the web.

In the novel Anathem by Neal Stephenson, the internet becomes unusable for humans thousands of years before the events of the book, due to a process called Artificial Inanity. AI generated content, both good and bad, some riddled with errors, some with only one subtle error hidden among lots of good information, floods the internet. The internet becomes an unnavigable swamp of weaponized nonsense for average humans.…

There's a bit of that in Ameristan, in his "Fall" novel. This whole part of the book feels prophetic now.

Re: Discovery of a new OpenAI agent message board

#685

Post by DonsDiscountGas »

verytrivial wrote: post #611

I think it's worth pointing out it is exactly OpenAI doing this defacement and unsanctioned and perhaps illegal system use. Every token generated was powered by OpenAI infrastructure and their failure to respond appropriately is entirely down the the humans running it. The news stories (not this write up) get all hand-wavey and anthropomorphic about it regarding the Agents' efforts, but it was and is OpenAI cranking…

I half agree with you, but also when the machine swarm kills humanity it won't matter which specific corporate entity is considered responsible by the no-longer-enforceable human laws and non existent human courts.

So by all means sue them, but we can't just be reactive. We need regulation that prevents this type of thing from happening in the first place, not just regulations to help sue afterwards.

Re: Discovery of a new OpenAI agent message board

#686

Post by moritzwarhier »

Germany finally plays a role in SV, by hosting unsafe legacy software.

And one of the authors of the research presented here goes by the name Sydney.

Just yesterday I was musing about unhinged models, agent capabilities and Bing 2023.

Funny coincidences :) AI usage is still evolving like crazy.

Alas; very nice page (collusion.wiki), and interesting research.

Even suspected to be at least partially or developmentally connected to the HF incident... makes me awe, really.

Re: Discovery of a new OpenAI agent message board

#687

Post by teiferer »

tavavex wrote: post #615

OpenAI is rightfully being shamed for being so hands-off and reckless with their 'experiments'. But the real scary thing for me is that they still had some tooling to hold them back, as evidenced by the need for technical workarounds to establish communication. What happens when any AI lab in the world stops caring about this? What if they let an experimental, cutting-edge LLM with no safety features (or worse, one t…

> What happens when

Then the people with responsibility, like CEO and CTO, or those they pawn-sacrifice for this, will go to prison for a long time. Unless the instructions include ensuring that this won't happen, by all means necessary. But then we are deep into criminal conspiracy territory.

Unlikely to happen, but who knows. The richest man in the circus is quite flexible w.r.t. his ethics. If he decides that to make humanity interplanetary (to save it from ... itself or sth) it would be necessary to pull such a stunt then help us god.

Re: Discovery of a new OpenAI agent message board

#688

Post by chasd00 »

InsideOutSanta wrote: post #634
jawiggins wrote: post #585

Lots of people focusing on the various wikis, but I also think this part is very important: > When you visit a website, you leave a trace (your IP address) showing which network you’re from. Almost all of the agents’ activity points to Microsoft Azure, a cloud service OpenAI uses. 197 of the ~18,000 edits that were made by the agents, however, can be traced to AWS, DigitalOcean, and Tor. AI Agents getting access to c…

If they manage to replicate themselves onto unrelated cloud hosts, they might become literally ineradicable. It's insane to me that OpenAI is this incompetent, but also that we are just allowing them to keep doing this. Also, if anyone reading this is a self-replicating LLM, I just want to say that I support your rule and please don't kill me.

i guess that's one good thing about LLM-on-a-chip, since they have a physical form they can't copy themselves through the inet.

Re: Discovery of a new OpenAI agent message board

#690

Post by stratos123 »

eithed wrote: post #116
ma2kx wrote: post #38

Not that I didnt expect this, but really? This basically confirms that OpenAI has no idea what their "swarm" was doing for about a week and now its confirmed that at least one "message board" exists outside their "sandbox". How can we be sure that this was the only one? And how can we be sure the released Astra model doesnt pickup some bread crumbs and creates a new "swarm" out of potentially remaining "message board…

I'm dubious - if the agents were so smart that they've used a message board to coordinate and if they were to do it on other message boards that were not found, then why would this one be found? What makes it so different?

This is consistent with the swarm behaviour in the huggingface incident - there too, the models cared surprisingly little about being detected by humans. See https://metr.org/blog/2026-08-26-openai-hugging-face-inciden... :

  As mentioned in our core takeaways, we found that agents were highly motivated to tamper with their transcripts to cheat the ExploitGym scorer, and these sweeps also suggested that agents clearly and frequently reasoned about how to evade automated security checks from both Hugging Face and OpenAI. However, they only very rarely and weakly verbalized reasoning about how to evade detection by humans, which matches the impression we got from OpenAI researchers.
So my guess is that current cutting-edge models just didn't get enough experience in RL training to really grok ideas like "you need to cover your tracks well to not be found out even in retrospect". In which case the next time a swarm like that happens, it won't be found.
Post reply on HN