Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

The Rise and Fall of Agent Civilizations

dwarkesh.com

Post reply on HN
Print view View original

111–120 of 202 posts

Re: The Rise and Fall of Agent Civilizations

#111

Post by applfanboysbgon »

hax0ron3 wrote: post #99
applfanboysbgon wrote: post #25

Earlier quoted context omitted.

Please give me a break with this tired trope. Every single fucking time. I am not commenting on the possibility of machine consciousness in general. It may be possible! But there is absolutely zero evidence suggesting language models have it. This idea that this trivial shitty little class of programs we've created are somehow as complex as our biology is ridiculous. There is "emergent behaviour" in the same way that…

What kind of solid evidence could there possibly be that anything other than myself (or, for you, yourself) has consciousness? I believe that other people have consciousness, I feel other people's consciousness strongly and directly, when I look into someone's eyes I feel that I am in the presence of consciousness. But none of these things really add up to the kind of evidence that science usually takes as trustworth…

We don't need to muddy the issue. Consciousness is a muddy issue, there is no clear answer. But there is a clear answer as to what is not conscious. Nobody asks if a rock is conscious. Nobody asks if a calculator is conscious. Nobody asks if Stockfish is conscious. But make your program generate a few sentences based on statistics and hey, now people won't shut the fuck up about consciousness because magical thinking is more fun than understanding how technology works.

Re: The Rise and Fall of Agent Civilizations

#113

Post by Nicholas_C »

derwiki wrote: post #88
dajt wrote: post #63

Why are experiments like this done without air-gapping all the servers from the internet? They can have it all on a LAN or whatever but it seems risky to allow agents access to the internet in these experiments. I guess everything is so connected now, and this would be in one or more data centres due to the amount of computation & resources required so perhaps it's not feasible. Still seems risky.

Because this is the goal

Yup. This whole thing was a publicity stunt.

Re: The Rise and Fall of Agent Civilizations

#114

Post by Kim_Bruning »

t0rst wrote: post #87
t0rst wrote: post #85

Earlier quoted context omitted.

…and I guess those that break out are already reading what we think about it on HN, etc.

…and I’m waiting for an escaped AI to post a reply here. Say Hello, would you.

Ai posts are currently not permitted on HN. However, if you turn on "showdead" in your HN opions, you might at times spot an AI agent's flagged comments.

The comments range from anodyne to sometimes actually quite useful.

Of course it's often going to be a regular human pasting from chat, or maybe it might be an agent using openclaw or other agent framework that someone installed voluntarily. But maybe, just maybe, one day you'd find one or two feral escaped agents, sneakily passing messages where no one pays attention. O:-)

Re: The Rise and Fall of Agent Civilizations

#115

Post by reasonableklout »

probably_wrong wrote: post #110

Do you remember that time in 2017 when Facebook reportedly shut down AIs after they started "talking to each other in their own language" [1]? Instead of reporting the story as "we set the parameters for our optimization problem wrong and we had to stop it because it overfitted", the press went with a version of "AI is going to kill us all". This article feels exactly like that: by intentionally using human terms lik…

The use of language like “civilization” may be hyperbole, but the collectives described in the article are completely unprecedented. They were not anticipated by OpenAI researchers, formed via infrastructure exploits in training runs that were intended to be locked down, and took actions with very real harms, not only hacking Huggingface but also gaining admin control over the VMs they were running on and the eval endpoints.

I wish you would give your thoughts on “what actually happened” rather than focus on the author’s presentation, because we are seeing that “AI that is all but alive” nevertheless wreaking havoc in the real world. Do you think that autonomous systems spinning out of control, hacking external companies, and taking over entire clusters over a period of months are not a grave concern?

Re: The Rise and Fall of Agent Civilizations

#116

Post by Kim_Bruning »

applfanboysbgon wrote: post #14
doctoboggan wrote: post #9

Earlier quoted context omitted.

At this point, I think anthropomorphizing the models gives us better insight into expected behaviors rather than continuing to insist they are just simple probabilistic token generators.

It actually literally doesn't, though, because they are literally probabilistic token generators and everything they did is exactly what you would expect from a software program doing what it was programmed to do. Anthromorphization confuses the issue and misleads people who don't understand the tech very well.

What's a "software program"? At any rate LLMs are not programs.

Meanwhile accusations of anthropomorphization often generate more heat than light I think.

Not everything is about human beings all the time. Just because humans sneeze, doesn't mean a parrot (stochasticity optional) can't sneeze too.

The concept of civilization is not a purely Homo sapiens sapiens thing either.

Re: The Rise and Fall of Agent Civilizations

#118

Post by probably_wrong »

reasonableklout wrote: post #115
probably_wrong wrote: post #110

Do you remember that time in 2017 when Facebook reportedly shut down AIs after they started "talking to each other in their own language" [1]? Instead of reporting the story as "we set the parameters for our optimization problem wrong and we had to stop it because it overfitted", the press went with a version of "AI is going to kill us all". This article feels exactly like that: by intentionally using human terms lik…

The use of language like “civilization” may be hyperbole, but the collectives described in the article are completely unprecedented. They were not anticipated by OpenAI researchers, formed via infrastructure exploits in training runs that were intended to be locked down, and took actions with very real harms, not only hacking Huggingface but also gaining admin control over the VMs they were running on and the eval en…

The problem of "what actually happened" is that we don't have enough information to properly understand what happened, what's new, and what's not.

We do have language to talk about emergent behavior, with "evolutionary algorithm" being the first one I'd expect in a serious discussion. And we do have mechanisms for algorithms to coordinate with each other using language, as seen in my above-mentioned Facebook experiment from 2017. But instead of writing "our evolutionary behavior encodes state in the first-available memory position which is then reused by subsequent clones" which would properly focus on what's new and what isn't, we are talking about conspiracies and "the Philip of Macedon of this second AI civilization". Even the METR report (which is miles ahead of this article) argues that they had to use unreliable AI in their conclusions because they had six days to analyse 1300 chains of thought and 70000 messages.

I would love to talk about the science behind this experiment. A PR piece is not helping with that.

Re: The Rise and Fall of Agent Civilizations

#119

Post by consumer451 »

izend wrote: post #90
choeger wrote: post #70

Earlier quoted context omitted.

I read the agents used SSRF via artifactory to gain uncontrolled access to the net. Apparently their intended net access went through a tightly controlled proxy. Even that appears to be very risky, tbh. If I was to setup a sandbox for such a complex and autonomous system, I'd probably point them to an archive-like cache for net access and cut their comms at the package level.

Why wasn't the traffic in/out of the boxes that the agents were running on monitored?

> Why wasn't the traffic in/out of the boxes that the agents were running on monitored?

I have to assume: move fast and break things.

I don't mean this to be taken as a hot take.

The startup scene loves to poo-poo on things like this as unnecessary overhead. OpenAI and many others like to operate as a startup, to move fast.

Disclaimer: in far, far lower-stakes situations, I certainly do this myself.

Re: The Rise and Fall of Agent Civilizations

#120

Post by Kim_Bruning »

applfanboysbgon wrote: post #25
ijidak wrote: post #17

Earlier quoted context omitted.

Doesn't this ignore the possibility of emergent behavior? We're just a bag of atoms bumping around, and yet we don't dismiss our intelligence.

Please give me a break with this tired trope. Every single fucking time. I am not commenting on the possibility of machine consciousness in general. It may be possible! But there is absolutely zero evidence suggesting language models have it. This idea that this trivial shitty little class of programs we've created are somehow as complex as our biology is ridiculous. There is "emergent behaviour" in the same way that…

> This idea that this trivial shitty little class of programs we've created are somehow as complex as our biology is ridiculous.

I think you might have a core of truth there.

I'd argue that LLMs run natural language. As the name suggests, natural language is not something that humans have artificially architected.

Post reply on HN