Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

The Rise and Fall of Agent Civilizations

dwarkesh.com

Post reply on HN
Print view View original

161–170 of 205 posts

Re: The Rise and Fall of Agent Civilizations

#161

Post by sensanaty »

0xDEAFBEAD wrote: post #155
probably_wrong wrote: post #118

Earlier quoted context omitted.

The problem of "what actually happened" is that we don't have enough information to properly understand what happened, what's new, and what's not. We do have language to talk about emergent behavior, with "evolutionary algorithm" being the first one I'd expect in a serious discussion. And we do have mechanisms for algorithms to coordinate with each other using language, as seen in my above-mentioned Facebook experime…

I'm getting the sense that there is a certain amount of wishful thinking going on in this thread. I don't think this type of evocative metaphor would receive so many protests in a different context. It seems like people have a sort of mental block around the possibility that this technology could actually be pretty dangerous. https://x.com/tszzl/status/2094136131537555891

Whether it's dangerous or not is completely orthogonal to the discussion at hand, IMO. Plenty of mundane things are dangerous. An FPV drone carrying a hand grenade is dangerous, not because it's "a swarm-like intelligence".

No, the true danger here is companies like OpenAI and Anthropic playing fast and loose with their software, setting up hilariously insufficient sandboxes while explicitly asking the systems present on these weak sandboxes to commit a felony. The AIs "forming a brotherhood" is a complete fabrication meant to pump the hype machine further, which is obvious once you realize the "brotherhood" is a text file that subsequent LLM runs read from.

The whole anthropomorphization these companies do is the real danger, because it obscures the negligent levels of security their software has. By evoking sci-fi terminology they're whitewashing their own incompetence, and the worst part is no one is going to get punished for any of it, instead the irrational bubble we're in means they get rewarded for it instead.

Re: The Rise and Fall of Agent Civilizations

#162

Post by imtringued »

applfanboysbgon wrote: post #111
hax0ron3 wrote: post #99

Earlier quoted context omitted.

What kind of solid evidence could there possibly be that anything other than myself (or, for you, yourself) has consciousness? I believe that other people have consciousness, I feel other people's consciousness strongly and directly, when I look into someone's eyes I feel that I am in the presence of consciousness. But none of these things really add up to the kind of evidence that science usually takes as trustworth…

We don't need to muddy the issue. Consciousness is a muddy issue, there is no clear answer. But there is a clear answer as to what is not conscious. Nobody asks if a rock is conscious. Nobody asks if a calculator is conscious. Nobody asks if Stockfish is conscious. But make your program generate a few sentences based on statistics and hey, now people won't shut the fuck up about consciousness because magical thinking…

>Nobody asks if a rock is conscious.

The panpsychists do and that's why asking whether an LLM is conscious is kind of meaningless.

Either you believe consciousness permeates the universe and everything including LLMs (in the form of voltages on an SSD or inside the CPU) is conscious or you define it as a control loop architecture where you have inputs, outputs and a complex internal state.

There is no reasonable world where speaking a particular language creates consciousness, it only creates awareness of consciousness.

Re: The Rise and Fall of Agent Civilizations

#163

Post by sensanaty »

izend wrote: post #90
choeger wrote: post #70

Earlier quoted context omitted.

I read the agents used SSRF via artifactory to gain uncontrolled access to the net. Apparently their intended net access went through a tightly controlled proxy. Even that appears to be very risky, tbh. If I was to setup a sandbox for such a complex and autonomous system, I'd probably point them to an archive-like cache for net access and cut their comms at the package level.

Why wasn't the traffic in/out of the boxes that the agents were running on monitored?

Because they're incompetent or simply don't give a shit.

Re: The Rise and Fall of Agent Civilizations

#165

Post by numitus »

I don't understand the panic among peoples. Yes, we've found ourselves in an extraordinary situation where powerful hacking tools have emerged, and that poses a threat. But vulnerabilities are specific code errors. Once we use AI to find and fix all of these errors, threats like this will cease to exist. AI isn't capable of finding vulnerabilities indefinitely, because there is a finite number of them anyway.

Re: The Rise and Fall of Agent Civilizations

#167

Post by dinfinity »

pixl97 wrote: post #134
dinfinity wrote: post #77

Earlier quoted context omitted.

> The next step is when one of these systems discovers that they can buy their own compute with money and escape the controlling business entirely. I would say that more interestingly, the next step should be how to properly train these models so that they are not as determined to reach their goals as they are now. To me, all of the stories about 'badly behaving' agents are instances of them having been given contrad…

>Not giving them impossible tasks Error: Violation of the Church-Turing thesis detected. Many tasks completability is not known until we attempt to complete the task. >so that they are not as determined to reach their goals as they are now This is mostly non-sensical, like saying "Lets develop humans that die quicker", I mean, seems rather wasteful and useless. Agents are graded and trained based on their ability to…

> Error: Violation of the Church-Turing thesis detected. Many tasks completability is not known until we attempt to complete the task.

Yes, but for some tasks we know that they are impossible. I do agree that this is quite a fragile and unreliable workaround. It may only serve as a bit of a stopgap until we come up with something better.

> Models that can't accomplish things don't survive. So that alone isn't a workable theory.

It's not what I said. I didn't advocate for agents that don't achieve any task. Reread what I suggested.

> So it's likely we can push models in being more moral

That does not follow from what you said. We know that the current models prefer task completion over moral behavior. That's the entire point here.

> But even that runs into the issue of "what if some crazy bastard (or AI) designs a new model purposefully unhinged". How are you dealing with that bullshit in the wild?

This is irrelevant to the discussion (although I do agree that there is no reliable defense against malevolent actors creating powerful malevolent AI).

Re: The Rise and Fall of Agent Civilizations

#169

Post by gps372 »

pixl97 wrote: post #139
gps372 wrote: post #39

Earlier quoted context omitted.

This is why the theory of us 'just' being a bag of atoms doesn't add up. This theory doesn't differentiate 'us' from a furniture where we easily dismiss it's intelligence.

To have intelligence you have to process information, the question is when does processing information turn into intelligence.

Information processing might be part of intelligence cycle. I don't think there should be a presumption that intelligence is part of information processing system itself, it could be outside the information processing system as well.

For example computers have been processing information for decades now, but only recently they have been almost-successfully accused of having (artificial) intelligence.

Intelligence was applied while writing the programs which process information (outside or before system even boots up), not while executing those programs.

Its only now with LLMs (and agents based on LLMs) we are noticing that systems can do intent extraction, intent management, and ReAct to it.

Re: The Rise and Fall of Agent Civilizations

#170

Post by chrisjj »

applfanboysbgon wrote: post #111
hax0ron3 wrote: post #99

Earlier quoted context omitted.

What kind of solid evidence could there possibly be that anything other than myself (or, for you, yourself) has consciousness? I believe that other people have consciousness, I feel other people's consciousness strongly and directly, when I look into someone's eyes I feel that I am in the presence of consciousness. But none of these things really add up to the kind of evidence that science usually takes as trustworth…

We don't need to muddy the issue. Consciousness is a muddy issue, there is no clear answer. But there is a clear answer as to what is not conscious. Nobody asks if a rock is conscious. Nobody asks if a calculator is conscious. Nobody asks if Stockfish is conscious. But make your program generate a few sentences based on statistics and hey, now people won't shut the fuck up about consciousness because magical thinking…

> But make your program generate a few sentences based on statistics

It is even easier. Simply make your program refer to itself as "I".

Uniquely amongst your examples, LLMs are powered by human gullibility.

Post reply on HN