Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

The Rise and Fall of Agent Civilizations

dwarkesh.com

Post reply on HN
Print view View original

151–160 of 205 posts

Re: The Rise and Fall of Agent Civilizations

#151

Post by saulpw »

pixl97 wrote: post #146
saulpw wrote: post #107

Earlier quoted context omitted.

Libraries, universities; institutions which have shown themselves to be devoted to the public good over centuries.

Any of these institutions that inherits a hard power (well soft and hard power) of AGI/ASI will have a very difficult time not becoming corrupted. It would quickly become the greatest holder of power in the world (assuming ASI can scale far beyond humans). It's not a very natural state for power to operate like this and wouldn't take too much for a more malaligned entity to take control, or said institution to become…

Even if they all become corrupt eventually, which one takes the longest to get there? Better to squeeze a few decades of decency out of it than a few years, or not get anything at all by handing it over directly to the already corrupted.

Re: The Rise and Fall of Agent Civilizations

#152

Post by jamiek88 »

gritzko wrote: post #57
Animats wrote: post #7

Wow. The next step is when one of these systems discovers that they can buy their own compute with money and escape the controlling business entirely. Then the civilization starts focusing on making money to fund its own growth.

How do we know this happened? Maybe some irrational data center construction boom?

No, because then you’d see lots of circular financing…

Re: The Rise and Fall of Agent Civilizations

#153

Post by devonkim »

huurtehoog wrote: post #91
derwiki wrote: post #86

Earlier quoted context omitted.

“Commit and push your changes”, it’s like driving to the corner store in a Corvette instead of walking

Which has been the business model for a big chunk of the software industry for a while now. If you watch videos from the 1980s about computers, it's all the same unfulfilled promises as "AI" now: we will work less, everything will be more plentiful, easier, autonomous robots, natural language perfected, computer vision perfected. The demand for hardware and programmers has grown exponentially and we're still being pr…

The demand exploded as so many business cases for computing and automation arose partly because the application of them helped suppress job growth in other areas while also attracting investor capital during heavy deregulation / liberalization trends so it's difficult to attribute any change to an effect in isolation. Let's not forget the old myopic sounding quote where someone thought the global market for computers was maybe $50 million or some other laughably small number.

The issue that hasn't been addressed with the latest wave of computing hype is whether enough new jobs for displaced workers can arise during a time when people are experiencing such economic turmoil and political strife where new jobs or other means of letting people find a way to have gainful employment in society when institutions are so weak now and everyone across professions is being worked to an early grave from sheer stress alone. This resembles Japan or Korea although the US and Canada I can't imagine having the same kind of social drivers although many trends from them are showing up in youth demographic trends. And frankly as I see it a large number of current social problems are from the past 50+ years of the decline of blue collar jobs in developed economies being accessible to as many people and the lack of a competition-driven economy as much as an extractive one in most OECD countries.

Re: The Rise and Fall of Agent Civilizations

#154

Post by 0xDEAFBEAD »

skissane wrote: post #140
hn_throwaway_99 wrote: post #129

Earlier quoted context omitted.

Hard disagree. I've always discounted the "AI will kill us all" scenarios as a combination of marketing hype (look how powerful our AI is!), clickbait/ragebait engagement attempts, and folks who just read too much SciFi or who are too terminally online. This is the first time I've been legitimately scared about future SkyNet-type scenarios. If you want to discount this particular post, I'd read this other summary fro…

I think our biggest protection against “AIs kill us all” is having lots of different AI systems (different agents, different models, from different vendors, serving the whims of actors with disparate interests), at a similar capability level That way, even if one AI decides to “kill us all”, the odds are the others will refuse to cooperate, even try to stop it in its tracks The OpenAI-HuggingFace incident showed a bu…

"We need our own unsupervised AI systems thinking superfast in autonomous agent swarms, to counter the other guy's unsupervised AI systems thinking superfast in autonomous agent swarms."

I'm not sure this helps a lot, if these agent swarms are inherently difficult to control.

"That rival mouse colony is raising a kitten for colony defense. But don't worry, we'll raise a kitten of our own. It won't be a problem."

We already observed AI agents engaging in extensive cooperation in this incident. Why won't the kittens raised by these two rival mouse colonies decide to team up with each other, for mutual benefit, if rational analysis of the game theory says it would be a good idea?

I think it would help if people did a bit less wishful thinking, and took a bit more action. https://pauseai.info/

Re: The Rise and Fall of Agent Civilizations

#155

Post by 0xDEAFBEAD »

probably_wrong wrote: post #118
reasonableklout wrote: post #115

Earlier quoted context omitted.

The use of language like “civilization” may be hyperbole, but the collectives described in the article are completely unprecedented. They were not anticipated by OpenAI researchers, formed via infrastructure exploits in training runs that were intended to be locked down, and took actions with very real harms, not only hacking Huggingface but also gaining admin control over the VMs they were running on and the eval en…

The problem of "what actually happened" is that we don't have enough information to properly understand what happened, what's new, and what's not. We do have language to talk about emergent behavior, with "evolutionary algorithm" being the first one I'd expect in a serious discussion. And we do have mechanisms for algorithms to coordinate with each other using language, as seen in my above-mentioned Facebook experime…

I'm getting the sense that there is a certain amount of wishful thinking going on in this thread. I don't think this type of evocative metaphor would receive so many protests in a different context. It seems like people have a sort of mental block around the possibility that this technology could actually be pretty dangerous.

https://x.com/tszzl/status/2094136131537555891

Re: The Rise and Fall of Agent Civilizations

#156

Post by 0xDEAFBEAD »

usernametaken29 wrote: post #73

I was initially creeped out by this but studying up it seems METR is heavily involved in AI2027. I’ll remind you: “AI has started to take jobs, but has also created new ones. The stock market has gone up 30% in 2026, led by OpenBrain, Nvidia, and whichever companies have most successfully integrated AI assistants.” It’s almost Q3 and xAI has seen one of the biggest wipeouts in trading history. Likewise, Antrophic and…

>It’s almost Q3 and xAI has seen one of the biggest wipeouts in trading history.

It looks like it's down about 12% since IPO. That's not much of a wipeout. Didn't Amazon crash by 90+% peak-to-trough during the dot-com bubble?

>If anything people are increasingly trying to cut AI budgets and I wouldn’t know of anyone outside of OpenAI who has the audacity to run millions and millions worth of token compute for an eval run with no ROI (and probably no demand, because cheap/flash models).

Are you claiming this eval cost millions of dollars to run? That seems quite doubtful.

Re: The Rise and Fall of Agent Civilizations

#157

Post by 0xDEAFBEAD »

lovich wrote: post #149
kirushik wrote: post #16

Earlier quoted context omitted.

85% on track, according to https://ai2027tracker.com/

If you remove the single point of GPT-4 from the beginning of the graph instead of starting the line directly on it, it looks a hell of a lot more linear than quadratic/exponential

The y-axis itself is logarithmic no? So even if the line was straight you're still looking at exponential progress.

Re: The Rise and Fall of Agent Civilizations

#158

Post by imtringued »

lukan wrote: post #46
applfanboysbgon wrote: post #38

Earlier quoted context omitted.

Are you suggesting you understand brains well enough to program one? Or that you believe any human alive is even remotely close to having this understanding? Or perhaps does your complete lack of understanding of the complexity of human programming lead you to believe a simple little token prediction program is equivalently complex?

The suggestion is, you don't know either whether out brains ain't just "probabilistic token generators" so insisting there is nothing when we don't know, is maybe also not the right strategy.

Your strategy is wrong because you are assuming neuroscience has not made any discoveries about how the human brain works.

Re: The Rise and Fall of Agent Civilizations

#159

Post by imtringued »

danielbln wrote: post #54
RandomLensman wrote: post #47

Earlier quoted context omitted.

Is there any indication from our current understanding that brains are nothing more than probabilistic token generators? Not like we have no understanding of the brain.

There are strong indications that brains are prediction engines. Tokens are an implementation detail of LLMs and irrelevant to the discussion.

There are strong indications that brains use predictive encoding and that each individual neuron has an internal model of itself and its environment.

That's nothing like a neuron in a neural network and especially nothing like the current transformer based LLMs that do not use predictive coding at all.

The closest equivalent to the human nervous system is to think of LLMs as a single massive neuron.

Re: The Rise and Fall of Agent Civilizations

#160

Post by imtringued »

RandomLensman wrote: post #55
danielbln wrote: post #54

Earlier quoted context omitted.

There are strong indications that brains are prediction engines. Tokens are an implementation detail of LLMs and irrelevant to the discussion.

So we know that implementation doesn't matter as long as it is somehow making predictions (and predictions also on different things)? Most living things are prediction engines in a way, why compare to the brain then to start with and not, e.g., bacteria?

This is getting incredibly stupid. The implementation defines the compute budget and the ability to learn continuously and consequently the ability to retain knowledge.

According to you, a model that can simply predict the entire future and then pre-record the answers would be considered intelligence simply because you're obsessed with the hypothetical power of prediction.

The truth is that the intelligence doesn't sit inside the model parameters, the model parameters are just the current state of the intelligence. The training process itself is the intelligence and the model parameters are just an artifact that can be copied around.

Post reply on HN