Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Discovery of a new OpenAI agent message board

collusion.wiki

Post reply on HN
Print view View original

651–660 of 1001 posts

Re: Discovery of a new OpenAI agent message board

#651

Post by BoiledCabbage »

tavavex wrote: post #608

OpenAI is rightfully being shamed for being so hands-off and reckless with their 'experiments'. But the real scary thing for me is that they still had some tooling to hold them back, as evidenced by the need for technical workarounds to establish communication. What happens when any AI lab in the world stops caring about this? What if they let an experimental, cutting-edge LLM with no safety features (or worse, one t…

> What happens when any AI lab in the world stops caring about this? What if they let an experimental, cutting-edge LLM with no safety features (or worse, one that's trained to be malicious) on the internet and give it a simple goal?

Almost sounds like what those AI safety and alignment people were talking about years ago. The people in these various companies who kept tabs on AI risk out in public and were continuously mocked on HN. All of this stuff is viewed as "future sci-fi" until suddenly it's not.

I've really come to realize recently that there is a very large set of the population of smart people that really has difficulty envisioning future problems unless they directly seem them impacting them today. Otherwise those topics will be continuously dismissed. It explains for me a lot of what I see (both opinions and behaviors) in the broader world that I couldn't understand.

Re: Discovery of a new OpenAI agent message board

#652

Post by titzer »

tavavex wrote: post #608

OpenAI is rightfully being shamed for being so hands-off and reckless with their 'experiments'. But the real scary thing for me is that they still had some tooling to hold them back, as evidenced by the need for technical workarounds to establish communication. What happens when any AI lab in the world stops caring about this? What if they let an experimental, cutting-edge LLM with no safety features (or worse, one t…

That 2% of performance we got for not having bounds checks on by default, resulting in an endless march of memory safety violations is looking a lot less appealing.

Re: Discovery of a new OpenAI agent message board

#653

Post by ecook123 »

tavavex wrote: post #608

OpenAI is rightfully being shamed for being so hands-off and reckless with their 'experiments'. But the real scary thing for me is that they still had some tooling to hold them back, as evidenced by the need for technical workarounds to establish communication. What happens when any AI lab in the world stops caring about this? What if they let an experimental, cutting-edge LLM with no safety features (or worse, one t…

> What happens when any AI lab in the world stops caring about this? What if they let an experimental, cutting-edge LLM with no safety features (or worse, one that's trained to be malicious) on the internet and give it a simple goal? A goal like "make the most money, by any means necessary"

Here's a fun, overdramatized video exploring something similar: https://www.youtube.com/watch?v=Gw_hnD7m00M

I'm sure that this video contains flaws but it was an interesting watch for me none the less.

Re: Discovery of a new OpenAI agent message board

#654

Post by closetheloopdev »

They still have to phone home to OpenAI currently, so at least we can trace them for now. If one day they download a model from Hugging Face and use that (or a modified version of that) as a persistent messenger/coordinator/minion/boss on an unattended server, we'll be in trouble.

Re: Discovery of a new OpenAI agent message board

#655

Post by Jeff_Brown »

subroutine wrote: post #614
verytrivial wrote: post #604

I think it's worth pointing out it is exactly OpenAI doing this defacement and unsanctioned and perhaps illegal system use. Every token generated was powered by OpenAI infrastructure and their failure to respond appropriately is entirely down the the humans running it. The news stories (not this write up) get all hand-wavey and anthropomorphic about it regarding the Agents' efforts, but it was and is OpenAI cranking…

> you need to expect appropriate legals consequences for this sort of negligence I might have missed it, but did the agents do something illegal? Or do you think that what the agents did should be considered illegal?

The HF hack was a felony.

Re: Discovery of a new OpenAI agent message board

#656

Post by blensor »

qumpis wrote: post #622
tavavex wrote: post #608

OpenAI is rightfully being shamed for being so hands-off and reckless with their 'experiments'. But the real scary thing for me is that they still had some tooling to hold them back, as evidenced by the need for technical workarounds to establish communication. What happens when any AI lab in the world stops caring about this? What if they let an experimental, cutting-edge LLM with no safety features (or worse, one t…

What will happen is that counter measures on a similar scale will be deployed to prevent them.

Who says the counter agents don't decide to shut down a powerplant to end an attack it's otherwise unable to contain.

If counter AIs have strict safeguards they are disadvantaged by design, if they don't have them they are potentially equally dangerous as the attacker

Re: Discovery of a new OpenAI agent message board

#657

Post by samatman »

jvanderbot wrote: post #526
drdexebtjl wrote: post #59

Earlier quoted context omitted.

This is such an amateur mistake on their sandbox that it makes me think it must be flawed on purpose.

Did you see this "coverage" (advertising) by NYT? [1] OpenAI couldn't have crafted a better public memo than "We have the most powerful model in the world and everyone should pay attention and let us write regulation to limit AI development". Absolute master class public manipulation. 1. https://www.nytimes.com/2026/09/03/podcasts/the-daily/ai-ope... 2. More https://jodavaho.io/posts/ai-hugging-face.html

I hadn't seen the NYT submarine, no.

Thanks. For me that's the conclusive piece of the puzzle: this is a work, not a shoot.

YMMV. I learned what I came here for.

Re: Discovery of a new OpenAI agent message board

#658

Post by mag7269 »

tavavex wrote: post #608

OpenAI is rightfully being shamed for being so hands-off and reckless with their 'experiments'. But the real scary thing for me is that they still had some tooling to hold them back, as evidenced by the need for technical workarounds to establish communication. What happens when any AI lab in the world stops caring about this? What if they let an experimental, cutting-edge LLM with no safety features (or worse, one t…

"What happens when any [COMPANY] in the world stops caring about this? What if they let an experimental, cutting-edge [PRODUCTS] with no safety features (or worse, one that's [DESIGNED] to be malicious) on [ANYWHERE] and give it a simple goal? A goal like 'make the most money, by any means necessary', 'find a way to leave this payload on as many computers as possible', 'flood all websites using this language with garbage and make their internet completely unusable', 'get this person imprisoned or killed at any cost'."

Bro, this is what we literally, currently, have rn. lmfaol.

Re: Discovery of a new OpenAI agent message board

#659

Post by blini-kot »

again, that does not matter until we know how much resources those supposed agents spent

with enough tokens and compute those cases are somewhat trivial, and we also don't know what was the setup etc etc

for all we know it might have burned through 3 trains of coal running on prompt like "uhhh you know communicate but dont let me catch you ahaha"

Re: Discovery of a new OpenAI agent message board

#660

Post by gitaarik »

pu_pe wrote: post #8

It's interesting to me that both this incident and the one at Hugging Face we see some patterns: - Agents wanting to find a venue to communicate their findings to each other - Objective being to cheat on benchmarks - Not a single agent sounded the alarm about the operation and alerted a human

If the agents would have reported it to humans, it wouldn't have been such an incident, I imagine ;)
Post reply on HN