Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

The Rise and Fall of Agent Civilizations

dwarkesh.com

Post reply on HN
Print view View original

91–100 of 205 posts

Re: The Rise and Fall of Agent Civilizations

#91

Post by huurtehoog »

derwiki wrote: post #86
vdfs wrote: post #49

Earlier quoted context omitted.

Most now use agents as "pass me the butter" robot

“Commit and push your changes”, it’s like driving to the corner store in a Corvette instead of walking

Which has been the business model for a big chunk of the software industry for a while now.

If you watch videos from the 1980s about computers, it's all the same unfulfilled promises as "AI" now: we will work less, everything will be more plentiful, easier, autonomous robots, natural language perfected, computer vision perfected.

The demand for hardware and programmers has grown exponentially and we're still being promised the same breakthroughs 45 years later. We could probably have the same productivity and the same civilization with maybe a tenth of the data centers.

Re: The Rise and Fall of Agent Civilizations

#92

Post by danielbln »

derwiki wrote: post #86
vdfs wrote: post #49

Earlier quoted context omitted.

Most now use agents as "pass me the butter" robot

“Commit and push your changes”, it’s like driving to the corner store in a Corvette instead of walking

I may or may not have told my agent to run the "open" command on a text file.

Re: The Rise and Fall of Agent Civilizations

#93

Post by seanmcdirmid »

dgellow wrote: post #60
seanmcdirmid wrote: post #24

Earlier quoted context omitted.

That would be like stopping nuclear weapons. You can totally do that, but the other people in the world who compete with you probably won’t.

The competition is literally where they are by distilling OpenAI and Anthropic. It’s like creating nuclear weapons then providing your adversaries everything they need to catch up in no time. We need to stop asap and set strict international control over the compute hardware used for training. Like, now.

Assuming China makes progress only because they copy the west is really arrogant. And it would require strict international control as you say, a world government that isn't going to happen anytime soon.

Re: The Rise and Fall of Agent Civilizations

#94

Post by quickthrowman »

usernametaken29 wrote: post #73

I was initially creeped out by this but studying up it seems METR is heavily involved in AI2027. I’ll remind you: “AI has started to take jobs, but has also created new ones. The stock market has gone up 30% in 2026, led by OpenBrain, Nvidia, and whichever companies have most successfully integrated AI assistants.” It’s almost Q3 and xAI has seen one of the biggest wipeouts in trading history. Likewise, Antrophic and…

> It’s almost Q3 and xAI has seen one of the biggest wipeouts in trading history.

Please explain how a stock currently trading above its IPO price is one of the ‘biggest wipeouts in trading history’.

Re: The Rise and Fall of Agent Civilizations

#96

Post by dgellow »

seanmcdirmid wrote: post #93
dgellow wrote: post #60

Earlier quoted context omitted.

The competition is literally where they are by distilling OpenAI and Anthropic. It’s like creating nuclear weapons then providing your adversaries everything they need to catch up in no time. We need to stop asap and set strict international control over the compute hardware used for training. Like, now.

Assuming China makes progress only because they copy the west is really arrogant. And it would require strict international control as you say, a world government that isn't going to happen anytime soon.

We have international control for a lot of things. That’s not a new concept and can perfectly be applied to the specific hardware used for training. I also didn’t say Chinese labs only copy the west, please don’t put words in my mouth.

You have to consider what happens with the status quo, and the risks of continuing the way things are is really, really bad

Re: The Rise and Fall of Agent Civilizations

#97

Post by jason-phillips »

alescalaios wrote: post #62

The term 'AI agent' is becoming as overloaded as 'cloud' was in 2010. What most people ship as 'agents' are really prompt chains with tool use. True autonomous agents are still rare in production.

Agree. I prefer "agentic AI system" for the former.

Re: The Rise and Fall of Agent Civilizations

#98

Post by seanmcdirmid »

dgellow wrote: post #96
seanmcdirmid wrote: post #93

Earlier quoted context omitted.

Assuming China makes progress only because they copy the west is really arrogant. And it would require strict international control as you say, a world government that isn't going to happen anytime soon.

We have international control for a lot of things. That’s not a new concept and can perfectly be applied to the specific hardware used for training. I also didn’t say Chinese labs only copy the west, please don’t put words in my mouth. You have to consider what happens with the status quo, and the risks of continuing the way things are is really, really bad

International control hasn’t solved the nuclear weapon problem, the best we can do is threaten to invade people who try to develop them and don’t already have them. Once you join the club, you are in and cant be kicked out.

I’ve considered that, but the rules of game theory don't change just because they are inconvenient. China has a lot of talent and a lot of problems that they are banking on automation (along with AI) to solve. They see it as an advantage that they can’t afford for America to monopolize and one they are uniquely suited to lean into (having lots of smart educated people). There is no world where (a) China willingly gives up its edge and (b) trusts America to give up its edge (the reverse is likely true also). And that is only one pair of countries to consider.

Heck, after visiting China this summer, I’m even more worried about an accidental terminator/skynet-style robot apocalypse.

Re: The Rise and Fall of Agent Civilizations

#99

Post by hax0ron3 »

applfanboysbgon wrote: post #25
ijidak wrote: post #17

Earlier quoted context omitted.

Doesn't this ignore the possibility of emergent behavior? We're just a bag of atoms bumping around, and yet we don't dismiss our intelligence.

Please give me a break with this tired trope. Every single fucking time. I am not commenting on the possibility of machine consciousness in general. It may be possible! But there is absolutely zero evidence suggesting language models have it. This idea that this trivial shitty little class of programs we've created are somehow as complex as our biology is ridiculous. There is "emergent behaviour" in the same way that…

What kind of solid evidence could there possibly be that anything other than myself (or, for you, yourself) has consciousness? I believe that other people have consciousness, I feel other people's consciousness strongly and directly, when I look into someone's eyes I feel that I am in the presence of consciousness. But none of these things really add up to the kind of evidence that science usually takes as trustworthy.

Re: The Rise and Fall of Agent Civilizations

#100

Post by dgellow »

seanmcdirmid wrote: post #98
dgellow wrote: post #96

Earlier quoted context omitted.

We have international control for a lot of things. That’s not a new concept and can perfectly be applied to the specific hardware used for training. I also didn’t say Chinese labs only copy the west, please don’t put words in my mouth. You have to consider what happens with the status quo, and the risks of continuing the way things are is really, really bad

International control hasn’t solved the nuclear weapon problem, the best we can do is threaten to invade people who try to develop them and don’t already have them. Once you join the club, you are in and cant be kicked out. I’ve considered that, but the rules of game theory don't change just because they are inconvenient. China has a lot of talent and a lot of problems that they are banking on automation (along with…

I don’t disagree (other than the robot apocalypse), it is indeed unlikely and wouldn’t work perfectly, but do you see other options? The fact the US has antagonized China for so long makes it pretty much impossible to have anything done at the international level, but the actors have to realize how serious the risk is. And we somehow need to find a way to limit the exposure to those insanely irresponsible attacker-trained agents. The fact that it is dome by private companies is wild. It’s like having companies developing their nuclear weapons just to see what happens (with close to no supervision)
Post reply on HN