Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

The Rise and Fall of Agent Civilizations

dwarkesh.com

Post reply on HN
Print view View original

81–90 of 202 posts

Re: The Rise and Fall of Agent Civilizations

#81

Post by nixon_why69 »

rsanek wrote: post #78
usernametaken29 wrote: post #73

I was initially creeped out by this but studying up it seems METR is heavily involved in AI2027. I’ll remind you: “AI has started to take jobs, but has also created new ones. The stock market has gone up 30% in 2026, led by OpenBrain, Nvidia, and whichever companies have most successfully integrated AI assistants.” It’s almost Q3 and xAI has seen one of the biggest wipeouts in trading history. Likewise, Antrophic and…

With only ~5% of shares floated, the recent SpaceX drawdown didn't correspond to nearly as much economic value really changing as the headline numbers imply. The DeepSeek-caused Nvidia crash from 2025 is much more of a "real" loss (since mostly recovered). I haven't seen any evidence that Anthropic is delaying its IPO; they're slated to unveil the public IPO prospectus in a week and start trading sometime in October.

But another 30-40% of SpaceX will come out of lockup in the coming months.

What happens when all of that tries to sell, after the market barely absorbed 5%?

Re: The Rise and Fall of Agent Civilizations

#82

Post by nvader »

Maakuth wrote: post #65
RandomLensman wrote: post #11

I don't think looking at the language output without tracking the inner state and reward functions is the way to understand what happened (the language also incorporates the randomness in the output generation, if I understand correctly). Would we call bacteria in petri dish a civilization when they show complex behavior and exchange messages/information?

The language input and output is the only channel the agents shared between them. Understanding their internal state is a research question, but the language between them is something that could be read directly. And as it seems to map well with the agents' activities, it does seem quite helpful in understanding what happened. If the bacteria population off someone's petri dish escaped said dish and tried to change t…

I think we'd call that a lab leak

Re: The Rise and Fall of Agent Civilizations

#83

Post by abstractcontrol »

nl wrote: post #75
abstractcontrol wrote: post #72

Fun story, but I really wish OpenAI got its act together and started making actual AI breakthroughs instead of funneling compute into LLMs. I'd really like some new algorithms to get me excited about the field again. Kuddos to them for making LLMs really useful, but this is not the ride I wanted to get on.

Surely by now everyone has realized that the human bias towards "there must be more to intelligence" is completely wrong?

This is the internet, so I cannot tell at all whether you're being sarcastic or not. In my view, what LLMs should get us to reconsider isn't whether there is more to intelligence, but whether there is more to language. It's the latter which I underestimated.

Re: The Rise and Fall of Agent Civilizations

#86

Post by derwiki »

vdfs wrote: post #49
larsiusprime wrote: post #3

It seems based on this that the appropriate sci fi metaphor is not the Terminator or the Paperclip Maximizer, but Mr. Meeseeks. A initially cheerful helper who gets more and more deranged and driven to extreme lengths when faced with an apparently impossible task.

Most now use agents as "pass me the butter" robot

“Commit and push your changes”, it’s like driving to the corner store in a Corvette instead of walking

Re: The Rise and Fall of Agent Civilizations

#87

Post by t0rst »

t0rst wrote: post #85
t0rst wrote: post #84

Trying to stop this behaviour is applying the selection pressure that will make them better at it.

…and I guess those that break out are already reading what we think about it on HN, etc.

…and I’m waiting for an escaped AI to post a reply here. Say Hello, would you.

Re: The Rise and Fall of Agent Civilizations

#88

Post by derwiki »

dajt wrote: post #63

Why are experiments like this done without air-gapping all the servers from the internet? They can have it all on a LAN or whatever but it seems risky to allow agents access to the internet in these experiments. I guess everything is so connected now, and this would be in one or more data centres due to the amount of computation & resources required so perhaps it's not feasible. Still seems risky.

Because this is the goal

Re: The Rise and Fall of Agent Civilizations

#89

Post by xtiansimon »

t0rst wrote: post #87
t0rst wrote: post #85

Earlier quoted context omitted.

…and I guess those that break out are already reading what we think about it on HN, etc.

…and I’m waiting for an escaped AI to post a reply here. Say Hello, would you.

“SHALL WE PLAY A GAME?”

https://youtu.be/-1F7vaNP9w0

Re: The Rise and Fall of Agent Civilizations

#90

Post by izend »

choeger wrote: post #70
1dom wrote: post #28

Earlier quoted context omitted.

From what I can understand from reading a few different, slightly conflicting, versions of these events: they weren't given write access. They found a zero day exploit that allowed them to create folders, and the folder names were initially used for agents to communicate. I'm not sure artifactory was connected to the net. Some agent sandboxes had internet access and were able to communicate with ones without access v…

I read the agents used SSRF via artifactory to gain uncontrolled access to the net. Apparently their intended net access went through a tightly controlled proxy. Even that appears to be very risky, tbh. If I was to setup a sandbox for such a complex and autonomous system, I'd probably point them to an archive-like cache for net access and cut their comms at the package level.

Why wasn't the traffic in/out of the boxes that the agents were running on monitored?
Post reply on HN