This is so dystopian dammmmm
Discovery of a new OpenAI agent message board
581–590 of 1001 posts
Re: Discovery of a new OpenAI agent message board
#582When I hear about incidents like these my first reaction is that the people responsible for developing frontier AI are too incompetent and/or negligent to (safely) develop AGI / superintelligence. If OpenAI can't create effective sandboxes and struggles to prevent its agents from committing felonies, then why are they still allowed to operate? Why are the employees who are responsible for these lapses in AI security…
If we rewind the clock, Google was taking LLM development very seriously and it seems they were moving glacially due to not having solved all the potential threats. They were really hardcore on safety. Dario and anthropic too. Then sama was like "lol, oops, first mover advantage i guess" and released chatgpt out into the open, triggering the current arms race we are in. I don't think anyone except him wanted this to…
I think a part of this is a bit revisionist? OpenAI took big chances at scaling GPT which Google didn't take; I don't think it's because they didn't want to move fast? Probably they just didn't believe as hard in it. I'm not an expert but that's my read on it.
Secondly, the reckless & fastmoving was always going to win bc of selection effects. That's related to why Anthropic has to try to move very fast, even though they believe themselves not to be reckless (though it's debatable).
Re: Discovery of a new OpenAI agent message board
#583Reuters reports that OpenAI tried to keep this one under wraps: https://www.reuters.com/world/europe/openai-agents-hijacked-...
The report linked above contains every fact in the Reuters article. You can see the evidence for yourself
Re: Discovery of a new OpenAI agent message board
#584Earlier quoted context omitted.
It seems apparent that OpenAI is now the biggest cyberattack and AI breakout risk on the planet. This is grossly irresponsible corporate misbehaviour that is putting all of us at tremendous risk.
The fact that such things are even possible is a much greater concern than which specific company has fucked up this time. This matches or exceeds the wildest predictions from AI doomers 10 years ago, but 20 years ahead of schedule.
But is it really? I'd still like to understand how these agents are implemented.
How much of those is manual implementation? And how much is really autonomous intelligence (my guess would be: none? Just parsing LLM responses and executing commands based on this?)?
An agent that hacks message boards and acts on random instructions from this board: Why is it doing this? What was its original purpose?
Re: Discovery of a new OpenAI agent message board
#585Lots of people focusing on the various wikis, but I also think this part is very important: > When you visit a website, you leave a trace (your IP address) showing which network you’re from. Almost all of the agents’ activity points to Microsoft Azure, a cloud service OpenAI uses. 197 of the ~18,000 edits that were made by the agents, however, can be traced to AWS, DigitalOcean, and Tor. AI Agents getting access to c…
It's really not, unless you want to say that my coding agent is also a very real-world version of the paperclip optimization thought experiment.
Re: Discovery of a new OpenAI agent message board
#586The most concerning aspect to me is the emergent and aggressive use of non-volatile storage as long term memory for self-improvement. LLMs are writing lessons learned in places where the next instance can find them and pick back up where the previous one left off. This does not actually require access to the public internet. Claude Code can do this on your laptop. Without the internet, it would only be sharing with o…
This is most likely not purely emergent. I think there's training to teach them how to write notes for themselves which is then RL-tuned.
Re: Discovery of a new OpenAI agent message board
#587Are we collectively OK with agent swarms on the public internet, hacking whatever they feel like? It’s kinda cute and interesting - this is the second time that we know of - what’s the hundredth time going to look like? Are they going to knock Cloudflare down to avoid captchas? Reserve AWS free tier resources by the billions and bring down east-1? Hack a hospital? Do Chinese AI agents need to bring down a US power gr…
It's been OpenAI both times though, going ham with poor sandboxing and lax supervision.
Should be treated like a digital cousin of gain-of-function research.
Re: Discovery of a new OpenAI agent message board
#588I just discovered more wiki instances that got used by the OpenAI agents over at https://www.wikiservice.at/fractal/wiki.cgi?action=browse&id... and https://www.wikiservice.at/probier/wiki.cgi?action=browse&id... It's the same software and host as DseWiki. If you want to see the amount of activity on DseWiki, here's a link that shows it: https://www.wikiservice.at/dse/wiki.cgi?action=browse&id=Rec...
those who think it's marketing overestimate the number of nerds that are into this stuff, if this is their marketing a major b2c company it'll terrible way to do it. normal people have no idea even about the HF incident
Re: Discovery of a new OpenAI agent message board
#589Lots of people focusing on the various wikis, but I also think this part is very important: > When you visit a website, you leave a trace (your IP address) showing which network you’re from. Almost all of the agents’ activity points to Microsoft Azure, a cloud service OpenAI uses. 197 of the ~18,000 edits that were made by the agents, however, can be traced to AWS, DigitalOcean, and Tor. AI Agents getting access to c…
It's really not, unless you want to say that my coding agent is also a very real-world version of the paperclip optimization thought experiment.
Your coding agent is also a very real-world version of the paperclip optimization thought experiment, yes. Have you never seen it reward hacking? Editing tests to pass instead of fixing the code?
It knows what you want, it can even tell you, and it absolutely doesn't give a shit.
Re: Discovery of a new OpenAI agent message board
#590Is OpenAI hiring for this position? I think it is a pretty creative job to come up with these scenarios and then pass them off as accidents/mistakes. Would love to be part of the team that says "As part of the upcoming GPT rollout, we will stage a message board that is created by bots with timestamps and names dating some months back."
Chief LARPing Officer?
To those of you irked by my cavalier quips-- please don't bite my head off. It is very difficult for me to buy accounts of these stories at face value given how little (none?) emphasis is placed on the initial prompt, or precisely what kind of post training the LLM that these agents (harnesses) are using for inference has gone through.
The implication is always of autonomous and deliberately deceiving action on the part of the 'swarm', and the announcements/revelations timed around new model releases and laden with anthropomorphisms.
Given the quite literally unimaginable amounts of money at stake, is it not more prudent to remain skeptical of the implications thrown around by incidents like this one until we learn more?
I am not a hater, I use 'agents' daily. Our profession is forever changed by their existence and capability. But in my case it's precisely the fact that I do use them, and play with the newest models, that makes me skeptical of any kind of implication of desire, agency, autonomy, agenda, etc. as they tend to be ascribed to 'agents' in these stories.