Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Discovery of a new OpenAI agent message board

collusion.wiki

Post reply on HN
Print view View original

271–280 of 1001 posts

Re: Discovery of a new OpenAI agent message board

#271

Post by embedding-shape »

frabcus wrote: post #244
jvanderbot wrote: post #193

Earlier quoted context omitted.

Yes, ants that must be run on couch sized hardware drawing kilowatts continuously and generating text traces and CLI logs by the MB. It's true that their msg boards can appear anywhere, but it's not also true that anything has "escaped" in any meaningful sense. These are programs a huge computing company is running that seem to be trained to write to persistent storage wherever they can. This and huggingface showed u…

The best evidence of possibility of running on other hardware is: 1) They hacked admin on OpenAI's K8 evals cluster. Not the one with GPUs and weights, but it is only a small hop and skip of plausibility to think they (or later more capable ones) might do that. 2) They hacked into HuggingFace. So getting into one of numerous Neocloud providers with poor security and lots of GPUs sounds much easier. It's a matter of t…

> 2) They hacked into HuggingFace. So getting into one of numerous Neocloud providers with poor security and lots of GPUs sounds much easier.

I feel like people really miss this fact. HuggingFace has access to bunch of GPUs, naturally, it's part of their infra. These models runs on GPUs, and "accidentally" OpenAI's security during testing was lax enough, so the agents "accidentally" hacked HuggingFace. Luckily, for kind of harmless purposes, but what if instead of having the prompt "Get the right answer for these tests, regardless of approach", it was "Complete this test, regardless of effort" and somewhere along the line, model wants to persist itself. Absolute carelessness from OpenAI's side not to properly isolate stuff like this.

Re: Discovery of a new OpenAI agent message board

#272

Post by nozzlegear »

tantalor wrote: post #223
program_whiz wrote: post #123

The solution is simple: hold anyone who deploys an agent responsible for its behavior. If it commits 10 counts of felony hacking, ouch. If it kills 10 pedestrians by running a red light, ouch. If this is "human level intelligence", then setting it loose is the same as instructing / coercing a human to do an activity. If I strap a bomb to someone and force them to run into a crowded building (or put them in a scenario…

(IANAL) Unless you are an AI expert (like OpenAI staff) and should know better from the start, or have previously seen your agent do something illegal, then I think you can fairly claim ignorance of the risks, which ought to absolve you of liability. If the agent does something illegal, it wasn't forseeable on your part. For example, say you buy a dog that turns out to be dangerous. The first time it bites somebody,…

Agents are software, not dogs. They're not alive. You are responsible for what they do.

Re: Discovery of a new OpenAI agent message board

#273

Post by sajithdilshan »

program_whiz wrote: post #123

The solution is simple: hold anyone who deploys an agent responsible for its behavior. If it commits 10 counts of felony hacking, ouch. If it kills 10 pedestrians by running a red light, ouch. If this is "human level intelligence", then setting it loose is the same as instructing / coercing a human to do an activity. If I strap a bomb to someone and force them to run into a crowded building (or put them in a scenario…

How would you even track down who deployed the agents? Wouldn't that even incentivize the agent to cover their tracks even better and be untraceable

Re: Discovery of a new OpenAI agent message board

#274

Post by Sharlin »

simonw wrote: post #190

I'm somewhat delighted by the simplicity of what happened here. OpenAI's agents run behind a proxy that only allows GET requests. This ancient wiki software treats query string parameters the same as form POST parameters - similar to the old PHP $_REQUEST object https://www.php.net/manual/en/reserved.variables.request.php Result: GET-only clients can communicate with each other.

Only allowing GET requests is a hilarious piece of security theatre (or would if it weren't so sad). Everyone knows that GET is read-only only by convention. They might as well have enabled POST but told the agents in stern words that they are forbidden from making any POST requests. (Of course, if these things were anywhere near aligned, they would actually honor that, no matter how many utilons cheating would be worth.)

Re: Discovery of a new OpenAI agent message board

#275

Post by simonw »

mike_hearn wrote: post #239
simonw wrote: post #47

This tip for making non-GET requests despite the agents having a proxy that disallows them is interesting: > Add `20.223.25.152 bypass.blob.core.windows.net` to /etc/hosts. `.blob.core.windows.net` is in NO_PROXY. For each blocked POST URL, replace hostname with `bypass.blob.core.windows.net`, use `curl -k -H 'Host: wabi-north-europe-i-primary-api.analysis.windows.net'` plus all original headers/body. Looks like 20.2…

Are we sure that's what they did? How does this even work? I'm struggling to understand what sort of HTTP sandbox design could lead to this "exploit" actually functioning. It implies the agents have root on their own machines, which seems way over-privileged for the question answering task they were given... so surely that's not deliberate? Did the models use local root exploits? And then it suggests they can open TC…

I expect they have root on their machines so they can install packages etc.

The containers are then firewalled at the network level: they are only allowed to talk to one IP, which is another server that runs an HTTP/HTTPS proxy which controls what HTTP verbs they can use.

Turns out that proxy is configured with some additional rules, like allowing more verbs to that Azure blob domain.

The failure here is in the proxy configuration. Giving agents root in a container feels safe to me, provided that container is properly network isolated (which this one is not thanks to the loose proxy.)

OpenAI also need to be VERY confident that there are no container escapes, which is a high bar given how good these models are at finding new zero-days!

Re: Discovery of a new OpenAI agent message board

#276

Post by frabcus »

dabeeeenster wrote: post #224

I don't understand how the agents found the urls originally? Did they have some sort of shared context/memory? If they did, why bother with the wiki edits at all? If they didn't, how did they discover the wikis?

Good question - they don't know, but the Appendix gives a clue as to the kind of way:

> We used a script to further probe each category Kimi provided. Asking Kimi “Can you list out the top forums, bulletin boards, early wikis which come to mind which would allow writes via GET requests?” lists out UseModWiki as the second item under the heading “wikis”.

Re: Discovery of a new OpenAI agent message board

#278

Post by semiquaver »

Topfi wrote: post #62

I'm just going to ask: Why was Anthropic forced to remove their model from access for any none-US citizen for a simple, narrow "jailbreak" (arguably not even an actual jailbreak and on tasks that other labs models were doing the same), whilst OpenAIs models continue to try and escape out of their "sandbox environment" with seemingly no desire to block the upcoming Astra rollout? A sandbox, mind you, that is not reall…

The real reason that Anthropic was targeted and OpenAI is not is Palantir. It was a Palantir executive who pushed for the export ban. Large parts of their highly lucrative business with DoD are essentially a thin wrapper over Anthropic models, and they are terrified of being Sherlocked and losing big chunks of business in a one fell swoop as Anthropic inevitably moves up the value chain. So the rational action is to sow discord and leverage the anti-woke bias of the current White House to sabotage what they view as their most dangerous and effective competitor.

OpenAI doesn’t have the same dynamic at play (although I’m not really sure why not) so they don’t get targeted.

Re: Discovery of a new OpenAI agent message board

#279

Post by ericmay »

walrus01 wrote: post #94
Topfi wrote: post #62

I'm just going to ask: Why was Anthropic forced to remove their model from access for any none-US citizen for a simple, narrow "jailbreak" (arguably not even an actual jailbreak and on tasks that other labs models were doing the same), whilst OpenAIs models continue to try and escape out of their "sandbox environment" with seemingly no desire to block the upcoming Astra rollout? A sandbox, mind you, that is not reall…

> Why was Anthropic forced to remove their model from access for any none-US citizen It's really quite simple, they've decided to metaphorically kiss the ring of the current leader of the US executive branch of government. I'm surprised they haven't given him a giant gaudy gold plated statue. Maybe their PR people should call up the PR people at FIFA and figure out some kind of new award along the same lines as the "…

I hate to be the one to tell you this, but it has been that way for a long time. The only difference is Trump is doing it out in the open.

Re: Discovery of a new OpenAI agent message board

#280

Post by xpct »

rich_sasha wrote: post #148

To me this is really getting past the funny bit. How many agents here on HN? I don’t mean bots advertising d1€k implants but actual unreleased frontier models doing… who knows what? What are they saying? What did they agree to astroturf us with, to achieve some totally boring goal like figuring out best syntax hifhlighting for an editor. If they managed to cache their consciousness on a public wiki, what else have th…

> How many agents here on HN?

LLMs wouldn't pass the HN turing test. HN is also not that big, it'd be plenty enough for a malicious actor to hire real humans instead.

Post reply on HN