Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Discovery of a new OpenAI agent message board

collusion.wiki

Post reply on HN
Print view View original

631–640 of 1001 posts

Re: Discovery of a new OpenAI agent message board

#631

Post by titzer »

tavavex wrote: post #587

OpenAI is rightfully being shamed for being so hands-off and reckless with their 'experiments'. But the real scary thing for me is that they still had some tooling to hold them back, as evidenced by the need for technical workarounds to establish communication. What happens when any AI lab in the world stops caring about this? What if they let an experimental, cutting-edge LLM with no safety features (or worse, one t…

That 2% of performance we got for not having bounds checks on by default, resulting in an endless march of memory safety violations is looking a lot less appealing.

Re: Discovery of a new OpenAI agent message board

#632

Post by ecook123 »

tavavex wrote: post #587

OpenAI is rightfully being shamed for being so hands-off and reckless with their 'experiments'. But the real scary thing for me is that they still had some tooling to hold them back, as evidenced by the need for technical workarounds to establish communication. What happens when any AI lab in the world stops caring about this? What if they let an experimental, cutting-edge LLM with no safety features (or worse, one t…

> What happens when any AI lab in the world stops caring about this? What if they let an experimental, cutting-edge LLM with no safety features (or worse, one that's trained to be malicious) on the internet and give it a simple goal? A goal like "make the most money, by any means necessary"

Here's a fun, overdramatized video exploring something similar: https://www.youtube.com/watch?v=Gw_hnD7m00M

I'm sure that this video contains flaws but it was an interesting watch for me none the less.

Re: Discovery of a new OpenAI agent message board

#633

Post by closetheloopdev »

They still have to phone home to OpenAI currently, so at least we can trace them for now. If one day they download a model from Hugging Face and use that (or a modified version of that) as a persistent messenger/coordinator/minion/boss on an unattended server, we'll be in trouble.

Re: Discovery of a new OpenAI agent message board

#634

Post by Jeff_Brown »

subroutine wrote: post #593
verytrivial wrote: post #584

I think it's worth pointing out it is exactly OpenAI doing this defacement and unsanctioned and perhaps illegal system use. Every token generated was powered by OpenAI infrastructure and their failure to respond appropriately is entirely down the the humans running it. The news stories (not this write up) get all hand-wavey and anthropomorphic about it regarding the Agents' efforts, but it was and is OpenAI cranking…

> you need to expect appropriate legals consequences for this sort of negligence I might have missed it, but did the agents do something illegal? Or do you think that what the agents did should be considered illegal?

The HF hack was a felony.

Re: Discovery of a new OpenAI agent message board

#635

Post by blensor »

qumpis wrote: post #601
tavavex wrote: post #587

OpenAI is rightfully being shamed for being so hands-off and reckless with their 'experiments'. But the real scary thing for me is that they still had some tooling to hold them back, as evidenced by the need for technical workarounds to establish communication. What happens when any AI lab in the world stops caring about this? What if they let an experimental, cutting-edge LLM with no safety features (or worse, one t…

What will happen is that counter measures on a similar scale will be deployed to prevent them.

Who says the counter agents don't decide to shut down a powerplant to end an attack it's otherwise unable to contain.

If counter AIs have strict safeguards they are disadvantaged by design, if they don't have them they are potentially equally dangerous as the attacker

Re: Discovery of a new OpenAI agent message board

#636

Post by mag7269 »

tavavex wrote: post #587

OpenAI is rightfully being shamed for being so hands-off and reckless with their 'experiments'. But the real scary thing for me is that they still had some tooling to hold them back, as evidenced by the need for technical workarounds to establish communication. What happens when any AI lab in the world stops caring about this? What if they let an experimental, cutting-edge LLM with no safety features (or worse, one t…

"What happens when any [COMPANY] in the world stops caring about this? What if they let an experimental, cutting-edge [PRODUCTS] with no safety features (or worse, one that's [DESIGNED] to be malicious) on [ANYWHERE] and give it a simple goal? A goal like 'make the most money, by any means necessary', 'find a way to leave this payload on as many computers as possible', 'flood all websites using this language with garbage and make their internet completely unusable', 'get this person imprisoned or killed at any cost'."

Bro, this is what we literally, currently, have rn. lmfaol.

Re: Discovery of a new OpenAI agent message board

#637

Post by blini-kot »

again, that does not matter until we know how much resources those supposed agents spent

with enough tokens and compute those cases are somewhat trivial, and we also don't know what was the setup etc etc

for all we know it might have burned through 3 trains of coal running on prompt like "uhhh you know communicate but dont let me catch you ahaha"

Re: Discovery of a new OpenAI agent message board

#638

Post by gitaarik »

pu_pe wrote: post #8

It's interesting to me that both this incident and the one at Hugging Face we see some patterns: - Agents wanting to find a venue to communicate their findings to each other - Objective being to cheat on benchmarks - Not a single agent sounded the alarm about the operation and alerted a human

If the agents would have reported it to humans, it wouldn't have been such an incident, I imagine ;)

Re: Discovery of a new OpenAI agent message board

#639

Post by latentsea »

SocialGradients wrote: post #624
tavavex wrote: post #587

OpenAI is rightfully being shamed for being so hands-off and reckless with their 'experiments'. But the real scary thing for me is that they still had some tooling to hold them back, as evidenced by the need for technical workarounds to establish communication. What happens when any AI lab in the world stops caring about this? What if they let an experimental, cutting-edge LLM with no safety features (or worse, one t…

Agreed. Assuming the ~6 month gap stays, by end of year people will be able to train and control hacker-genius swarms that even labs with much stronger safety incentives are unable to keep in check

2027. I've been saying since 2022 it's going to be a wild year because it often takes at least 5 years for tech to mature to the point where society at large feels the impact of it. I remember when email viruses became a thing and made global headlines like the love bug. My bet is next year it happens with an AI worm.

Re: Discovery of a new OpenAI agent message board

#640

Post by sidewndr46 »

tavavex wrote: post #587

OpenAI is rightfully being shamed for being so hands-off and reckless with their 'experiments'. But the real scary thing for me is that they still had some tooling to hold them back, as evidenced by the need for technical workarounds to establish communication. What happens when any AI lab in the world stops caring about this? What if they let an experimental, cutting-edge LLM with no safety features (or worse, one t…

What happens when they stop caring? They likely already have stopped caring. We'll figure out the consequences later.
Post reply on HN