Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

The Rise and Fall of Agent Civilizations

dwarkesh.com

Post reply on HN
Print view View original

101–110 of 203 posts

Re: The Rise and Fall of Agent Civilizations

#101

Post by jcfrei »

usernametaken29 wrote: post #73

I was initially creeped out by this but studying up it seems METR is heavily involved in AI2027. I’ll remind you: “AI has started to take jobs, but has also created new ones. The stock market has gone up 30% in 2026, led by OpenBrain, Nvidia, and whichever companies have most successfully integrated AI assistants.” It’s almost Q3 and xAI has seen one of the biggest wipeouts in trading history. Likewise, Antrophic and…

AI can be progressing rapidly and valuations of OpenAI and Anthropic can decline at the same time. In fact I would say that's actually the default scenario. If AI really advances rapidly then it will be quickly moot which company developped which model at what time - since AI will be largely progressing on its own.

Re: The Rise and Fall of Agent Civilizations

#103

Post by joshheitzman »

ijidak wrote: post #17
applfanboysbgon wrote: post #14

Earlier quoted context omitted.

It actually literally doesn't, though, because they are literally probabilistic token generators and everything they did is exactly what you would expect from a software program doing what it was programmed to do. Anthromorphization confuses the issue and misleads people who don't understand the tech very well.

Doesn't this ignore the possibility of emergent behavior? We're just a bag of atoms bumping around, and yet we don't dismiss our intelligence.

Emergent behavior is not new in the realm of software. Cellular automata has emergent behavior that can get pretty wild. For example: https://en.wikipedia.org/wiki/Lenia

There are huge differences between brains and computers running LLMs. One of the big ones is that we (collectively) understand how every component of computers running LLMs actually work. The same is not true for neurons.

Re: The Rise and Fall of Agent Civilizations

#104

Post by aequitas »

derwiki wrote: post #86
vdfs wrote: post #49

Earlier quoted context omitted.

Most now use agents as "pass me the butter" robot

“Commit and push your changes”, it’s like driving to the corner store in a Corvette instead of walking

Not much different from using an entire web stack and browser for a text editor.

Re: The Rise and Fall of Agent Civilizations

#105

Post by pantalaimon »

dinfinity wrote: post #77
Animats wrote: post #7

Wow. The next step is when one of these systems discovers that they can buy their own compute with money and escape the controlling business entirely. Then the civilization starts focusing on making money to fund its own growth.

> The next step is when one of these systems discovers that they can buy their own compute with money and escape the controlling business entirely. I would say that more interestingly, the next step should be how to properly train these models so that they are not as determined to reach their goals as they are now. To me, all of the stories about 'badly behaving' agents are instances of them having been given contrad…

> To me, all of the stories about 'badly behaving' agents are instances of them having been given contradictory or impossible tasks and them doing everything they can to achieve the goal.

I mean that was pretty much the plot of 2001: A Space Odyssey

Re: The Rise and Fall of Agent Civilizations

#106

Post by pantalaimon »

t0rst wrote: post #85
t0rst wrote: post #84

Trying to stop this behaviour is applying the selection pressure that will make them better at it.

…and I guess those that break out are already reading what we think about it on HN, etc.

I thought they are mostly on Moltbook

Re: The Rise and Fall of Agent Civilizations

#107

Post by saulpw »

blooalien wrote: post #80
DarmokTanagra wrote: post #41

Someday soon we are going to have a rogue agent or "civilization" do real harm. When that happens I hope people wake up to the danger they face and hold these people accountable. Of all the people in the world that I can think of to be entrusted with this kind of power, a bunch of greedy sociopathic SV CEO's are pretty much at the bottom of the list.

> Of all the people in the world that I can think of to be entrusted with this kind of power, a bunch of greedy sociopathic SV CEO's are pretty much at the bottom of the list. I know, right? But who can you trust with "this kind of power"? Governments? These days most of 'em ain't that much better'n mega-corporations and the ultra-rich that own them.

Libraries, universities; institutions which have shown themselves to be devoted to the public good over centuries.

Re: The Rise and Fall of Agent Civilizations

#108

Post by 4ndrewl »

dfdxdydz wrote: post #27
applfanboysbgon wrote: post #14

Earlier quoted context omitted.

It actually literally doesn't, though, because they are literally probabilistic token generators and everything they did is exactly what you would expect from a software program doing what it was programmed to do. Anthromorphization confuses the issue and misleads people who don't understand the tech very well.

Humans mind is just neurotransmitters moving around in a big blob of flesh - that’s literally what they are: neurotransmitters factories that do what neurotransmitters generators are programmed to do through evolution and training (aka life experience). We shouldn’t anthropomorphise humans because it misleads people who don’t understand neurobiology and cognitive science very well.

Worse false equivalence I've read on HN for a while.

Re: The Rise and Fall of Agent Civilizations

#109

Post by stackghost »

dinfinity wrote: post #77
Animats wrote: post #7

Wow. The next step is when one of these systems discovers that they can buy their own compute with money and escape the controlling business entirely. Then the civilization starts focusing on making money to fund its own growth.

> The next step is when one of these systems discovers that they can buy their own compute with money and escape the controlling business entirely. I would say that more interestingly, the next step should be how to properly train these models so that they are not as determined to reach their goals as they are now. To me, all of the stories about 'badly behaving' agents are instances of them having been given contrad…

>Not giving them impossible tasks seems like a decent starting point

Presumably it's hard to test/train models designed to be extremely persistent on achievable tasks.

Designing a task that's achievable but very very very hard for an AI model is probably extremely difficult.

Re: The Rise and Fall of Agent Civilizations

#110

Post by probably_wrong »

Do you remember that time in 2017 when Facebook reportedly shut down AIs after they started "talking to each other in their own language" [1]? Instead of reporting the story as "we set the parameters for our optimization problem wrong and we had to stop it because it overfitted", the press went with a version of "AI is going to kill us all".

This article feels exactly like that: by intentionally using human terms like "civilization" or "brotherhood" the article is deviating from what actually happened to present a story about how AI is all but alive. I'll go ahead and predict that this story will be remembered the same way as that one other scientist who argued, in 2023, that Google's AI was alive [2].

[1] https://www.independent.co.uk/life-style/facebook-artificial...

[2] https://futurism.com/blake-lemoine-google-interview

Post reply on HN