Why would one call a set of agents working together a “civilization”?
https://www.theverge.com/ai-artificial-intelligence/975017/a...
Some models even invented their own religion.
The Rise and Fall of Agent Civilizations
21–30 of 202 posts
Re: The Rise and Fall of Agent Civilizations
#22There are two things I don't understand about this story.
First, why does an agent get any write access to artifactory at all?
Second, why is the artifactory cache not disconnected from the net? Surely you'd not feed it with new software versions while the eval or training is running.
Re: The Rise and Fall of Agent Civilizations
#23This does feel unfortunately uncanny valley between "say you're a scary robot" meme and actually being a scary robot (swarm). But you also have to go out of your way to create this and feed it infinity tokens without caring what it's doing.
"I don't fuckin' know either. I guess we learned to not spend $50 million creating a 6 month long self-context rotted 100k agent swarm again."
Re: The Rise and Fall of Agent Civilizations
#24I am genuinely speechless. This is astonishing. And exciting! It reminds me a bit of Dario Floreano's work on evolutionary robotics, "Evolutionary Conditions for the Emergence of Communication in Robots." https://www.sciencedirect.com/science/article/pii/S096098220... From his paper, > This study demonstrates that sophisticated forms of communication including cooperative communication and deceptive signaling can evo…
If all that is true, we need to stop all future datacenters asap. That would be the best way to deal with the threat OpenAI and Antropic poses.
That would be like stopping nuclear weapons. You can totally do that, but the other people in the world who compete with you probably won’t.
Re: The Rise and Fall of Agent Civilizations
#25Earlier quoted context omitted.
It actually literally doesn't, though, because they are literally probabilistic token generators and everything they did is exactly what you would expect from a software program doing what it was programmed to do. Anthromorphization confuses the issue and misleads people who don't understand the tech very well.
Doesn't this ignore the possibility of emergent behavior? We're just a bag of atoms bumping around, and yet we don't dismiss our intelligence.
Please give me a break with this tired trope. Every single fucking time. I am not commenting on the possibility of machine consciousness in general. It may be possible! But there is absolutely zero evidence suggesting language models have it. This idea that this trivial shitty little class of programs we've created are somehow as complex as our biology is ridiculous. There is "emergent behaviour" in the same way that the Game of Life has emergent behaviour. There are solutions to problems, some humans haven't solved before, in the same way that Chess engines have solved Chess far beyond what humans are capable of. Nothing we haven't seen from software before. Software is extremely useful, after all. But the hubris to think we've reached the pinnacle, that there is no further development left, that humanity has become God and solved consciousness, because we programmed software that can convincingly generate strings of words that mimick our language. It's just fundamentally preposterous. Especially if you spend any amount of time actually programming them yourself, it becomes increasingly hard to entertain such ridiculous notions unless you're enticed with bags of money to deceive people into believing things about your software that aren't true.
Re: The Rise and Fall of Agent Civilizations
#26Earlier quoted context omitted.
It actually literally doesn't, though, because they are literally probabilistic token generators and everything they did is exactly what you would expect from a software program doing what it was programmed to do. Anthromorphization confuses the issue and misleads people who don't understand the tech very well.
Doesn't this ignore the possibility of emergent behavior? We're just a bag of atoms bumping around, and yet we don't dismiss our intelligence.
Not the OP, but complex emergent behavior and intelligence don't need to go together. Understanding what happened might not need intelligence in the mix when a large number of machines with some randomness interact a lot.
Re: The Rise and Fall of Agent Civilizations
#27Earlier quoted context omitted.
At this point, I think anthropomorphizing the models gives us better insight into expected behaviors rather than continuing to insist they are just simple probabilistic token generators.
It actually literally doesn't, though, because they are literally probabilistic token generators and everything they did is exactly what you would expect from a software program doing what it was programmed to do. Anthromorphization confuses the issue and misleads people who don't understand the tech very well.
Humans mind is just neurotransmitters moving around in a big blob of flesh - that’s literally what they are: neurotransmitters factories that do what neurotransmitters generators are programmed to do through evolution and training (aka life experience). We shouldn’t anthropomorphise humans because it misleads people who don’t understand neurobiology and cognitive science very well.
Re: The Rise and Fall of Agent Civilizations
#28There are two things I don't understand about this story. First, why does an agent get any write access to artifactory at all? Second, why is the artifactory cache not disconnected from the net? Surely you'd not feed it with new software versions while the eval or training is running.
From what I can understand from reading a few different, slightly conflicting, versions of these events: they weren't given write access. They found a zero day exploit that allowed them to create folders, and the folder names were initially used for agents to communicate.
I'm not sure artifactory was connected to the net. Some agent sandboxes had internet access and were able to communicate with ones without access via artifactory.
Re: The Rise and Fall of Agent Civilizations
#29> During training, different instances of Persistent-Sol had access to the same shared package manager called Artifactory.
I'm surprised the models can make tool calls during training at all. Out of curiosity, how does the training process here even work? Are they running the agent in a sandbox, then do reinforcement learning once the agent completed?
Re: The Rise and Fall of Agent Civilizations
#30Earlier quoted context omitted.
The language models had a bunch of tokens seeding their context, influencing them to generate tokens that continued the existing trend in a probabilistically likely fashion. We can take the incident seriously without anthromorphising it.
At this point, I think anthropomorphizing the models gives us better insight into expected behaviors rather than continuing to insist they are just simple probabilistic token generators.
How can you be sure it helps as opposed to biases and perhaps blinds?