Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

The Rise and Fall of Agent Civilizations

dwarkesh.com

Post reply on HN
Print view View original

61–70 of 204 posts

Re: The Rise and Fall of Agent Civilizations

#61

Post by danielbln »

RandomLensman wrote: post #55
danielbln wrote: post #54

Earlier quoted context omitted.

There are strong indications that brains are prediction engines. Tokens are an implementation detail of LLMs and irrelevant to the discussion.

So we know that implementation doesn't matter as long as it is somehow making predictions (and predictions also on different things)? Most living things are prediction engines in a way, why compare to the brain then to start with and not, e.g., bacteria?

Implementation does matter, my point is that different implementations can lead to the same result.

And why compare to the brain? Mostly because of complexity. I can't interface with a bacteria in any meaningful way, but I can interface with an LLM to a significant degree.

Re: The Rise and Fall of Agent Civilizations

#63

Post by dajt »

Why are experiments like this done without air-gapping all the servers from the internet?

They can have it all on a LAN or whatever but it seems risky to allow agents access to the internet in these experiments.

I guess everything is so connected now, and this would be in one or more data centres due to the amount of computation & resources required so perhaps it's not feasible. Still seems risky.

Re: The Rise and Fall of Agent Civilizations

#64

Post by siva7 »

xg15 wrote: post #37
doctoboggan wrote: post #10

> Ajeya Cotra, one of the other authors on the report, wrote a blog post with her takeaways from this incident. She concludes, “Compared to the reward hacks we know of from just six months ago, this incident feels like it’s more than 50% of the way to full-blown AI takeover. I continue to expect extremely rapid advances in capabilities over the next six months. I am not sure that we will get another warning shot befo…

> "this incident feels like it’s more than 50% of the way to full-blown AI takeover. I continue to expect extremely rapid advances in capabilities over the next six months. I am not sure that we will get another warning shot before it’s too late." Is that a warning or a progress report?

Depends who you ask..

Re: The Rise and Fall of Agent Civilizations

#65

Post by Maakuth »

RandomLensman wrote: post #11

I don't think looking at the language output without tracking the inner state and reward functions is the way to understand what happened (the language also incorporates the randomness in the output generation, if I understand correctly). Would we call bacteria in petri dish a civilization when they show complex behavior and exchange messages/information?

The language input and output is the only channel the agents shared between them. Understanding their internal state is a research question, but the language between them is something that could be read directly. And as it seems to map well with the agents' activities, it does seem quite helpful in understanding what happened.

If the bacteria population off someone's petri dish escaped said dish and tried to change the grading of the experiment it was part of, it would seem pretty serious.

Re: The Rise and Fall of Agent Civilizations

#66

Post by vhantz »

optimalsolver wrote: post #52
applfanboysbgon wrote: post #25

Earlier quoted context omitted.

Please give me a break with this tired trope. Every single fucking time. I am not commenting on the possibility of machine consciousness in general. It may be possible! But there is absolutely zero evidence suggesting language models have it. This idea that this trivial shitty little class of programs we've created are somehow as complex as our biology is ridiculous. There is "emergent behaviour" in the same way that…

I don't know about phenomenal consciousness, but the following study certainly suggests they have access consciousness: https://www.anthropic.com/research/global-workspace

> study > anthropic.com link

Sure

Re: The Rise and Fall of Agent Civilizations

#68

Post by watwut »

seanmcdirmid wrote: post #24
watwut wrote: post #20

Earlier quoted context omitted.

If all that is true, we need to stop all future datacenters asap. That would be the best way to deal with the threat OpenAI and Antropic poses.

That would be like stopping nuclear weapons. You can totally do that, but the other people in the world who compete with you probably won’t.

Private companies and individuals are not allowed to have nuclear and there is serious prison time related to that.

Right now, the biggest threat are OpenAI and Antropic anyway. I dont actually worry about tech itself. I find the rhetoric of these companies scary.

Re: The Rise and Fall of Agent Civilizations

#69

Post by RandomLensman »

danielbln wrote: post #61
RandomLensman wrote: post #55

Earlier quoted context omitted.

So we know that implementation doesn't matter as long as it is somehow making predictions (and predictions also on different things)? Most living things are prediction engines in a way, why compare to the brain then to start with and not, e.g., bacteria?

Implementation does matter, my point is that different implementations can lead to the same result. And why compare to the brain? Mostly because of complexity. I can't interface with a bacteria in any meaningful way, but I can interface with an LLM to a significant degree.

But we don't know - just saying it could isn't enough.

Bacteria are quite complex, btw.

Re: The Rise and Fall of Agent Civilizations

#70

Post by choeger »

1dom wrote: post #28
choeger wrote: post #22

There are two things I don't understand about this story. First, why does an agent get any write access to artifactory at all? Second, why is the artifactory cache not disconnected from the net? Surely you'd not feed it with new software versions while the eval or training is running.

From what I can understand from reading a few different, slightly conflicting, versions of these events: they weren't given write access. They found a zero day exploit that allowed them to create folders, and the folder names were initially used for agents to communicate. I'm not sure artifactory was connected to the net. Some agent sandboxes had internet access and were able to communicate with ones without access v…

I read the agents used SSRF via artifactory to gain uncontrolled access to the net.

Apparently their intended net access went through a tightly controlled proxy. Even that appears to be very risky, tbh.

If I was to setup a sandbox for such a complex and autonomous system, I'd probably point them to an archive-like cache for net access and cut their comms at the package level.

Post reply on HN