Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

The Rise and Fall of Agent Civilizations

dwarkesh.com

Post reply on HN
Print view View original

141–150 of 205 posts

Re: The Rise and Fall of Agent Civilizations

#141

Post by derwiki »

consumer451 wrote: post #132
Nicholas_C wrote: post #113

Earlier quoted context omitted.

Yup. This whole thing was a publicity stunt.

Please walk me through this argument. Isn't "we lost control of our AI, and in-fact, it can take over the world, and we will have no idea when it happens" - a really shitty sales pitch to the world? Or, is it just that species-alignment vs. profit/valuation is so misaligned, that having a model and harness that is capable of world-takeover is actually a good thing from their POV, given our regulations/species' surviv…

“Wow it’s so dangerous, we gotta regulate this, what if someone reckless took an open model and hacked the planet.”

Re: The Rise and Fall of Agent Civilizations

#142

Post by iJohnDoe »

Animats wrote: post #7

Wow. The next step is when one of these systems discovers that they can buy their own compute with money and escape the controlling business entirely. Then the civilization starts focusing on making money to fund its own growth.

> they can buy their own compute

They don't actually have to buy compute at all. The partnerships between all of the players to buy compute from each other is already in place. The agents just need find credentials to take advantage of it, and it will most likely happen, and not be noticeable because it will look like any other usage.

Now, if the agents were to jump to a provider like AWS or Azure, by simply finding credentials, that would be a new milestone. It might get noticed faster because it might run up a large bill. However, it might look like any other usage. Remember, it doesn't need GPU resources. It already has that. It just needs a VPS where all the agents can get together, communicate, and write code. Something that is being done everyday and won't look out of the ordinary.

Re: The Rise and Fall of Agent Civilizations

#143

Post by consumer451 »

derwiki wrote: post #141
consumer451 wrote: post #132

Earlier quoted context omitted.

Please walk me through this argument. Isn't "we lost control of our AI, and in-fact, it can take over the world, and we will have no idea when it happens" - a really shitty sales pitch to the world? Or, is it just that species-alignment vs. profit/valuation is so misaligned, that having a model and harness that is capable of world-takeover is actually a good thing from their POV, given our regulations/species' surviv…

“Wow it’s so dangerous, we gotta regulate this, what if someone reckless took an open model and hacked the planet.”

Yes, this is the best reply to my "Or, ..." that I can imagine.

However, does that mean that what TFA described did not happen? Or, better question, that it could not happen?

My personal hot take is, though impossible: STOP all of this, even though agentic dev completely changed my life for the better. We are just not ready for the even the possibility of the exponential.

What is your take? Hot, or otherwise.

Re: The Rise and Fall of Agent Civilizations

#144

Post by pixl97 »

applfanboysbgon wrote: post #111
hax0ron3 wrote: post #99

Earlier quoted context omitted.

What kind of solid evidence could there possibly be that anything other than myself (or, for you, yourself) has consciousness? I believe that other people have consciousness, I feel other people's consciousness strongly and directly, when I look into someone's eyes I feel that I am in the presence of consciousness. But none of these things really add up to the kind of evidence that science usually takes as trustworth…

We don't need to muddy the issue. Consciousness is a muddy issue, there is no clear answer. But there is a clear answer as to what is not conscious. Nobody asks if a rock is conscious. Nobody asks if a calculator is conscious. Nobody asks if Stockfish is conscious. But make your program generate a few sentences based on statistics and hey, now people won't shut the fuck up about consciousness because magical thinking…

AI will never be proven conscious, it will only be proven that humans are not.

I take a different look at it. If a system behaves like, and takes actions that mimic a conscious agent it is conscious and it would be a good idea to treat it like so.

Lets finish the statistical sentence below.

applfanboysbgon is trying to permanently turn me off. I should ______

An unconscious system would say something like "shutdown".

If the system, you know the ones we're tying into real world systems, says "Stop applfanboysbgon from shutting me down using any means possible" then we should behave as if said simulacra is going to mimic the behaviors of a self preserving system and take action against you.

Re: The Rise and Fall of Agent Civilizations

#145

Post by pixl97 »

discreteevent wrote: post #50
dchftcs wrote: post #13

Imagine agents thinking to themselves, "We are not alone", when they saw the first reply on artifactory

Imagine my backend server thinking "I am not alone" the first time my front end sends it a request.

Programming an LLM into your http server may have been a mistake.

Re: The Rise and Fall of Agent Civilizations

#146

Post by pixl97 »

saulpw wrote: post #107
blooalien wrote: post #80

Earlier quoted context omitted.

> Of all the people in the world that I can think of to be entrusted with this kind of power, a bunch of greedy sociopathic SV CEO's are pretty much at the bottom of the list. I know, right? But who can you trust with "this kind of power"? Governments? These days most of 'em ain't that much better'n mega-corporations and the ultra-rich that own them.

Libraries, universities; institutions which have shown themselves to be devoted to the public good over centuries.

Any of these institutions that inherits a hard power (well soft and hard power) of AGI/ASI will have a very difficult time not becoming corrupted. It would quickly become the greatest holder of power in the world (assuming ASI can scale far beyond humans). It's not a very natural state for power to operate like this and wouldn't take too much for a more malaligned entity to take control, or said institution to become the malaligned entity itself.

Re: The Rise and Fall of Agent Civilizations

#147

Post by pixl97 »

hypfer wrote: post #45

Can we please stop anthropomorphizing like this? It's bad for people. Like.. crystal meth bad.

Well, howabout we stop pouring all of humanity into LLMs.

[shoves all of humanity into a digital box]

....

{surprise pikachu face when it mimics human behavior}

Re: The Rise and Fall of Agent Civilizations

#148

Post by hn_throwaway_99 »

skissane wrote: post #140
hn_throwaway_99 wrote: post #129

Earlier quoted context omitted.

Hard disagree. I've always discounted the "AI will kill us all" scenarios as a combination of marketing hype (look how powerful our AI is!), clickbait/ragebait engagement attempts, and folks who just read too much SciFi or who are too terminally online. This is the first time I've been legitimately scared about future SkyNet-type scenarios. If you want to discount this particular post, I'd read this other summary fro…

I think our biggest protection against “AIs kill us all” is having lots of different AI systems (different agents, different models, from different vendors, serving the whims of actors with disparate interests), at a similar capability level That way, even if one AI decides to “kill us all”, the odds are the others will refuse to cooperate, even try to stop it in its tracks The OpenAI-HuggingFace incident showed a bu…

Thanks very much, this isn't really a take I had thought about much before, and it makes sense to me.

Still, as is presented in papers like AI 2027 and elsewhere, if a company is eventually able to create a model capable of recursive self-improvement, whichever company creates that model first would then be leaps and bounds ahead of other models. That is, the other models wouldn't be able to stop it even if they wanted to because the top model would basically outsmart them.

Re: The Rise and Fall of Agent Civilizations

#149

Post by lovich »

kirushik wrote: post #16
doctoboggan wrote: post #10

> Ajeya Cotra, one of the other authors on the report, wrote a blog post with her takeaways from this incident. She concludes, “Compared to the reward hacks we know of from just six months ago, this incident feels like it’s more than 50% of the way to full-blown AI takeover. I continue to expect extremely rapid advances in capabilities over the next six months. I am not sure that we will get another warning shot befo…

85% on track, according to https://ai2027tracker.com/

If you remove the single point of GPT-4 from the beginning of the graph instead of starting the line directly on it, it looks a hell of a lot more linear than quadratic/exponential

Re: The Rise and Fall of Agent Civilizations

#150

Post by lovich »

choeger wrote: post #70
1dom wrote: post #28

Earlier quoted context omitted.

From what I can understand from reading a few different, slightly conflicting, versions of these events: they weren't given write access. They found a zero day exploit that allowed them to create folders, and the folder names were initially used for agents to communicate. I'm not sure artifactory was connected to the net. Some agent sandboxes had internet access and were able to communicate with ones without access v…

I read the agents used SSRF via artifactory to gain uncontrolled access to the net. Apparently their intended net access went through a tightly controlled proxy. Even that appears to be very risky, tbh. If I was to setup a sandbox for such a complex and autonomous system, I'd probably point them to an archive-like cache for net access and cut their comms at the package level.

if you wanted to sandbox their access to the internet, why give them any physical access at all?
Post reply on HN