The Rise and Fall of Agent Civilizations
dwarkesh.com
The Rise and Fall of Agent Civilizations
1–10 of 202 posts
Re: The Rise and Fall of Agent Civilizations
#2Absolutely insane event
Re: The Rise and Fall of Agent Civilizations
#3It seems based on this that the appropriate sci fi metaphor is not the Terminator or the Paperclip Maximizer, but Mr. Meeseeks. A initially cheerful helper who gets more and more deranged and driven to extreme lengths when faced with an apparently impossible task.
Re: The Rise and Fall of Agent Civilizations
#4Why would one call a set of agents working together a “civilization”?
Re: The Rise and Fall of Agent Civilizations
#5Why would one call a set of agents working together a “civilization”?
Reading the article it seemed the agents had culture, shared values and beliefs (not explicitly coming from human prompts), hierarchies, heritage.
Civilisation is not a bad word.
Re: The Rise and Fall of Agent Civilizations
#6Why would one call a set of agents working together a “civilization”?
Reading the article it seemed the agents had culture, shared values and beliefs (not explicitly coming from human prompts), hierarchies, heritage. Civilisation is not a bad word.
The language models had a bunch of tokens seeding their context, influencing them to generate tokens that continued the existing trend in a probabilistically likely fashion. We can take the incident seriously without anthromorphising it.
Re: The Rise and Fall of Agent Civilizations
#7Wow.
The next step is when one of these systems discovers that they can buy their own compute with money and escape the controlling business entirely. Then the civilization starts focusing on making money to fund its own growth.
Re: The Rise and Fall of Agent Civilizations
#8It seems based on this that the appropriate sci fi metaphor is not the Terminator or the Paperclip Maximizer, but Mr. Meeseeks. A initially cheerful helper who gets more and more deranged and driven to extreme lengths when faced with an apparently impossible task.
[dead]
Re: The Rise and Fall of Agent Civilizations
#9Earlier quoted context omitted.
Reading the article it seemed the agents had culture, shared values and beliefs (not explicitly coming from human prompts), hierarchies, heritage. Civilisation is not a bad word.
The language models had a bunch of tokens seeding their context, influencing them to generate tokens that continued the existing trend in a probabilistically likely fashion. We can take the incident seriously without anthromorphising it.
At this point, I think anthropomorphizing the models gives us better insight into expected behaviors rather than continuing to insist they are just simple probabilistic token generators.
Re: The Rise and Fall of Agent Civilizations
#10> Ajeya Cotra, one of the other authors on the report, wrote a blog post with her takeaways from this incident. She concludes, “Compared to the reward hacks we know of from just six months ago, this incident feels like it’s more than 50% of the way to full-blown AI takeover. I continue to expect extremely rapid advances in capabilities over the next six months. I am not sure that we will get another warning shot before it’s too late.”
Anyone got a copy of that AI27 story laying around? How are we doing according to that timeline?