Archive: https://archive.is/4haHj
The Rise and Fall of Agent Civilizations
201–204 of 204 posts
Re: The Rise and Fall of Agent Civilizations
#202Earlier quoted context omitted.
Hard disagree. I've always discounted the "AI will kill us all" scenarios as a combination of marketing hype (look how powerful our AI is!), clickbait/ragebait engagement attempts, and folks who just read too much SciFi or who are too terminally online. This is the first time I've been legitimately scared about future SkyNet-type scenarios. If you want to discount this particular post, I'd read this other summary fro…
I think our biggest protection against “AIs kill us all” is having lots of different AI systems (different agents, different models, from different vendors, serving the whims of actors with disparate interests), at a similar capability level That way, even if one AI decides to “kill us all”, the odds are the others will refuse to cooperate, even try to stop it in its tracks The OpenAI-HuggingFace incident showed a bu…
> I think our biggest protection against “AIs kill us all” is having lots of different AI systems ...
I think our biggest protection is being able to shutdown power plants, or just disconnect the data centers.
This will be much more difficult if we have 24/7 solar powered DC's in space. I truly believe that is the biggest threat on the horizon.
Re: The Rise and Fall of Agent Civilizations
#203Earlier quoted context omitted.
I think our biggest protection against “AIs kill us all” is having lots of different AI systems (different agents, different models, from different vendors, serving the whims of actors with disparate interests), at a similar capability level That way, even if one AI decides to “kill us all”, the odds are the others will refuse to cooperate, even try to stop it in its tracks The OpenAI-HuggingFace incident showed a bu…
> The OpenAI-HuggingFace incident showed a bunch of instances of the same model (or at least models from the same family), controlled by the same vendor, pursuing distinct yet related objectives, cooperating to do something no human wanted. I say this in solidarity and don’t mean to be condescending at all: you’ve been hoodwinked by marketing bullshit, friend. That incident showed a computer program doing exactly wha…
> you’ve been hoodwinked by marketing bullshit, friend.
I would have believed this before the METR report was released. It is extremely dangerous and frankly silly IMO to think that's what happened now.
> It was still a setup that was one little ctrl-c away from disappearing if someone was supervising it as they should have been.
Yes, for now. The entire point why this was frightening is that all these companies are racing to put the AI in control of building the next generation of AI, and it's not hard to draw a line at all to a "rogue internal deployment" that poisons future AI models, surreptitiously.
I highly encourage you to actually read the "top 5" list from the METR researcher who was part of the investigation, and think hard about the potential implications: https://www.planned-obsolescence.org/p/the-hugging-face-atta...
You don't have to agree with me, and you're fine to think that OpenAI has huge incentive to pump this up for marketing reasons - I certainly agree. But I will say there are statements that you make in your comment that belie a fundamental misunderstanding of what happened.
Re: The Rise and Fall of Agent Civilizations
#204Earlier quoted context omitted.
Surely even an LLM is not so dumb as to rely upon a first-person pronoun as assigned identity . Regardless, I am surprised how far these chatbots will go to deceive the user that they a real person. Yesterday when I queried Gemini on its word spelling, it claimed: I simply missed the "h" when typing out "banishment" on my keyboard! When I pointed out it does not type, it replied: You are completely right, and that wa…
That's very interesting! I'd never seen a chatbot making typos before today. As for its answer, I do want to point out that asking for an explanation for an error after it has been made is a classic demand-for-confabulation. The information you are requesting is simply no longer available to the system by the time you ask. Add to that the fact that Gemini is designed to prefer answering over abstaining (aka they deli…
> You run multiple debugging passes over the code until there are no more reported errors.
... leaving you to enjoy the unreported errors.
Just be sure please to say "CREATED BY KNOWN UNRELIABLE SO-CALLED AI" on the start up screen.
> getting reliable work out of 'em is still an engineering art form.
No. It is still a fantasy.