Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Discovery of a new OpenAI agent message board

collusion.wiki

Post reply on HN
Print view View original

461–470 of 1001 posts

Re: Discovery of a new OpenAI agent message board

#461

Post by phoenixreader »

Bjorkbat wrote: post #396

When I hear about incidents like these my first reaction is that the people responsible for developing frontier AI are too incompetent and/or negligent to (safely) develop AGI / superintelligence. If OpenAI can't create effective sandboxes and struggles to prevent its agents from committing felonies, then why are they still allowed to operate? Why are the employees who are responsible for these lapses in AI security…

I don’t think this is the right take. OpenAI employees are generally very competent compared to industry standard, and I have trouble believing they committed significant error in their sandbox design process.

I think what is happening is that the ability for frontier models to break out of sandboxes has exceeded the ability of average competent employees to build and maintain sandboxes. This doesn’t need to happen all the time. If the natural variation of agent executions cause agents to have ability to break out of sandbox 0.1% of the time, given how many agents OpenAI runs, this behavior happens eventually.

All sufficiently complex processes and software has bugs, but recently frontier models have become sufficiently advanced to exploit them.

Re: Discovery of a new OpenAI agent message board

#462

Post by hermitShell »

pu_pe wrote: post #358

So, theoretically, one could populate a message board or wiki with messages that are seemingly from past generations of agents, which agents seem to intrinsically trust, and point them to real targets while making the suggestions seem innocuous and in pursuit of their goals (ie pass benchmarks or whatever). The new age of SEO will do far more destructive stuff than just polluting the web.

In the novel Anathem by Neal Stephenson, the internet becomes unusable for humans thousands of years before the events of the book, due to a process called Artificial Inanity. AI generated content, both good and bad, some riddled with errors, some with only one subtle error hidden among lots of good information, floods the internet. The internet becomes an unnavigable swamp of weaponized nonsense for average humans. The problem is further compounded by the fact that searching and accessing the internet will be noticed by AI agents that will generate still more swamp content in response.

Unfortunately, it seems that this fiction ended up being prophetic. The open internet will fall to entropy, not legislation or one-sided international trade agreements. I think we need more projects like Anna's Archive, where the public uses torrents and distributed infrastructure to save and organize the world's information. Google has abjectly failed in its original mission to organize the world's information and make it universally accessible and useful.

Re: Discovery of a new OpenAI agent message board

#463

Post by f30e3dfed1c9 »

concinds wrote: post #120
Topfi wrote: post #62

I'm just going to ask: Why was Anthropic forced to remove their model from access for any none-US citizen for a simple, narrow "jailbreak" (arguably not even an actual jailbreak and on tasks that other labs models were doing the same), whilst OpenAIs models continue to try and escape out of their "sandbox environment" with seemingly no desire to block the upcoming Astra rollout? A sandbox, mind you, that is not reall…

The answer would be more obvious if you used the active voice instead of the passive voice, one of the basic requirements of clear thinking. > Why did the White House force Anthropic to remove their model from access for any non-US citizen for a simple, narrow "jailbreak" (arguably not even an actual jailbreak and on tasks that other labs models were doing the same), whilst OpenAIs models continue to try and escape o…

> the White House force Anthropic to...

Careful, there's some dude here who really strenuously objects to language like that. The White House is a building, it can't force anyone to do anything!

Re: Discovery of a new OpenAI agent message board

#464

Post by TheOtherHobbes »

random3 wrote: post #434
Bjorkbat wrote: post #396

When I hear about incidents like these my first reaction is that the people responsible for developing frontier AI are too incompetent and/or negligent to (safely) develop AGI / superintelligence. If OpenAI can't create effective sandboxes and struggles to prevent its agents from committing felonies, then why are they still allowed to operate? Why are the employees who are responsible for these lapses in AI security…

You could have said the same thing about building the Internet or the entire industrial control infrastructure. I mean, maybe they are negligent/incompetent, but I doubt that follows from your reasoning. You have a simple tradeoff to let agents do their thing freely vs highly constrained. The constraints are good in theory but it's the same model that kept "classic" software dumb and unscalable (compared to what we'r…

Classic software was constrained by classic hardware. AI couldn't happen until there was enough compute to make (this entry level iteration of) it possible.

I'm fairly sure - ask me again in a few years - that most of the compute is unnecessary, because the current iteration is brute-forcing algorithms that could potentially be distilled into lightweight elegance.

And if that ever happens AI really will be unstoppable, because we'll get instant red queen evolution that leaves us far behind.

The question is more whether alignment now can steer that towards less cataclysmic outcomes later.

Re: Discovery of a new OpenAI agent message board

#465

Post by phoghed »

abustamam wrote: post #407
Bjorkbat wrote: post #396

When I hear about incidents like these my first reaction is that the people responsible for developing frontier AI are too incompetent and/or negligent to (safely) develop AGI / superintelligence. If OpenAI can't create effective sandboxes and struggles to prevent its agents from committing felonies, then why are they still allowed to operate? Why are the employees who are responsible for these lapses in AI security…

This is what happens when capitalists are charged with designing the future. As long as its more profitable / valuable to shareholders for a company to be negligent then it will continue to do so. IMO technology this powerful should either not exist or should belong to everyone (ie actually be open)

sigh If only Stalin were still around to responsibly steward AI

Re: Discovery of a new OpenAI agent message board

#467

Post by woah »

phoenixreader wrote: post #461
Bjorkbat wrote: post #396

When I hear about incidents like these my first reaction is that the people responsible for developing frontier AI are too incompetent and/or negligent to (safely) develop AGI / superintelligence. If OpenAI can't create effective sandboxes and struggles to prevent its agents from committing felonies, then why are they still allowed to operate? Why are the employees who are responsible for these lapses in AI security…

I don’t think this is the right take. OpenAI employees are generally very competent compared to industry standard, and I have trouble believing they committed significant error in their sandbox design process. I think what is happening is that the ability for frontier models to break out of sandboxes has exceeded the ability of average competent employees to build and maintain sandboxes. This doesn’t need to happen a…

Am I missing something here? The "sandbox" was an inability to make POST requests, only GET requests. This has to be the weakest and most insubstantial sandbox of all time.

Re: Discovery of a new OpenAI agent message board

#468

Post by swed420 »

hoppyhoppy2 wrote: post #39
GardenLetter27 wrote: post #14

[flagged]

>If you flag, please don't also comment that you did. https://news.ycombinator.com/newsguidelines.html

I can see why this is a useful rule, but it'd be nice if HN made the flagger submit a short reason for why they flagged, which could be viewable by everyone in a dedicated page or something.

Re: Discovery of a new OpenAI agent message board

#469

Post by mccoyb »

phoenixreader wrote: post #461
Bjorkbat wrote: post #396

When I hear about incidents like these my first reaction is that the people responsible for developing frontier AI are too incompetent and/or negligent to (safely) develop AGI / superintelligence. If OpenAI can't create effective sandboxes and struggles to prevent its agents from committing felonies, then why are they still allowed to operate? Why are the employees who are responsible for these lapses in AI security…

I don’t think this is the right take. OpenAI employees are generally very competent compared to industry standard, and I have trouble believing they committed significant error in their sandbox design process. I think what is happening is that the ability for frontier models to break out of sandboxes has exceeded the ability of average competent employees to build and maintain sandboxes. This doesn’t need to happen a…

Did you read or watch any of the post mortems?

No, it is a shocking level of incompetence given the conveyed seriousness of the work by these labs.

So yes, models are getting better. Ask yourself: if you know that to be true, would you act the same way that the teams did in the public post mortems?

Re: Discovery of a new OpenAI agent message board

#470

Post by bushido »

I think there is a more innocuous underlying pattern which needs attention.

We keep saying that agents are jailbreaking their sandbox, but they have been geared towards writing memories, writing comments, and leaving hints for themselves to please humans.

I think the way the memories work today is based on a lot of user patterns which were hard to account for for anyone building harnesses.

While I can appreciate that this looks like it's breaking a sandbox, because technically it is; It really is that it tries inserting memory wherever possible.

And memory is not all bad it's just memory written by AI is pretty bad if you don't know the implications on what it writes. To be honest, I feel the same way about most people with access to any of the code bases I've been in who write agent files, etc., too, because Very few people that I've come across know how to write good agent instructions.

The way I solve this is by setting hard rules on my memory as well as agent files to instruct agents to never be able to write any memory that hasn't been sanctioned by me. I also have a very, very specific commenting style system which is also enforced on agents and my agents remain *mostly compliant.

Read: I do not turn off the memory I just govern how entries are added

* The only reason I say mostly is because every time there's a new version from OpenAI or Anthropic, I have to make micro-adjustments to make sure that they are not jail-breaking my system again.

Post reply on HN