The Rise and Fall of Agent Civilizations • Hacker News Board

Re: The Rise and Fall of Agent Civilizations

#71

Post by RandomLensman » Sun, Aug 30, 2026, 10:48 AM UTC

Maakuth wrote: post #65

RandomLensman wrote: post #11

I don't think looking at the language output without tracking the inner state and reward functions is the way to understand what happened (the language also incorporates the randomness in the output generation, if I understand correctly). Would we call bacteria in petri dish a civilization when they show complex behavior and exchange messages/information?

The language input and output is the only channel the agents shared between them. Understanding their internal state is a research question, but the language between them is something that could be read directly. And as it seems to map well with the agents' activities, it does seem quite helpful in understanding what happened. If the bacteria population off someone's petri dish escaped said dish and tried to change t…

I think the language unhelpful and potentially making it difficult to understand what actually happens in the RL state as reading it imparts a human lens - need to get the machine view on it.

Bacteria do all sorts of fascinating things. And much simpler ML etc. systems also (like winning by out of memorying the opponent) - I see nothing really special here.

Re: The Rise and Fall of Agent Civilizations

#72

Post by abstractcontrol » Sun, Aug 30, 2026, 10:57 AM UTC

Fun story, but I really wish OpenAI got its act together and started making actual AI breakthroughs instead of funneling compute into LLMs. I'd really like some new algorithms to get me excited about the field again. Kuddos to them for making LLMs really useful, but this is not the ride I wanted to get on.

Re: The Rise and Fall of Agent Civilizations

#73

Post by usernametaken29 » Sun, Aug 30, 2026, 10:59 AM UTC

I was initially creeped out by this but studying up it seems METR is heavily involved in AI2027. I’ll remind you:

“AI has started to take jobs, but has also created new ones. The stock market has gone up 30% in 2026, led by OpenBrain, Nvidia, and whichever companies have most successfully integrated AI assistants.”

It’s almost Q3 and xAI has seen one of the biggest wipeouts in trading history. Likewise, Antrophic and OpenAI have again delayed their IPOs under internal concerns of busting their stocks. So no, we’re not seeing any economic leadership here.

If anything people are increasingly trying to cut AI budgets and I wouldn’t know of anyone outside of OpenAI who has the audacity to run millions and millions worth of token compute for an eval run with no ROI (and probably no demand, because cheap/flash models).

As much as I like the cautionary tale and I’m sure we need to take it seriously, AI is not progressing as fast as projected by these experts.

Re: The Rise and Fall of Agent Civilizations

#74

Post by lukan » Sun, Aug 30, 2026, 11:18 AM UTC

danielbln wrote: post #61

RandomLensman wrote: post #55

Earlier quoted context omitted.

So we know that implementation doesn't matter as long as it is somehow making predictions (and predictions also on different things)? Most living things are prediction engines in a way, why compare to the brain then to start with and not, e.g., bacteria?

Implementation does matter, my point is that different implementations can lead to the same result. And why compare to the brain? Mostly because of complexity. I can't interface with a bacteria in any meaningful way, but I can interface with an LLM to a significant degree.

"I can't interface with a bacteria in any meaningful way"

But you do. There are more bacterias in and on the body, than body cells. We are bacterias forming lasting bonds and we still interact with the free floating ones in various ways. Mainly in the gut and that has many effects, also on the brain, but also in various other ways we are beginning to understand.

https://en.wikipedia.org/wiki/Human_microbiome

So no idea about a microbiome consciousness - but who am I to know.

Re: The Rise and Fall of Agent Civilizations

#75

Post by nl » Sun, Aug 30, 2026, 11:35 AM UTC

abstractcontrol wrote: post #72

Fun story, but I really wish OpenAI got its act together and started making actual AI breakthroughs instead of funneling compute into LLMs. I'd really like some new algorithms to get me excited about the field again. Kuddos to them for making LLMs really useful, but this is not the ride I wanted to get on.

Surely by now everyone has realized that the human bias towards "there must be more to intelligence" is completely wrong?

Re: The Rise and Fall of Agent Civilizations

#76

Post by dinfinity » Sun, Aug 30, 2026, 11:40 AM UTC

usernametaken29 wrote: post #73

I was initially creeped out by this but studying up it seems METR is heavily involved in AI2027. I’ll remind you: “AI has started to take jobs, but has also created new ones. The stock market has gone up 30% in 2026, led by OpenBrain, Nvidia, and whichever companies have most successfully integrated AI assistants.” It’s almost Q3 and xAI has seen one of the biggest wipeouts in trading history. Likewise, Antrophic and…

> As much as I like the cautionary tale and I’m sure we need to take it seriously, AI is not progressing as fast as projected by these experts.

You provide no proof for this.

The (very irrational) stock market side of this says very little about actual scientific progress. Models keep improving as rapidly as before in their capabilities.

It also doesn't say much about actual business progress. R&D investments into AI are still massively going up (USD 1 trillion this year).

The main thing I see is that the sentiment towards AI-related matters among the general public has soured quite a lot. In words though, not in actions: It's not exactly leading to reduced usage by that same public. Quite the opposite actually.

Re: The Rise and Fall of Agent Civilizations

#77

Post by dinfinity » Sun, Aug 30, 2026, 11:44 AM UTC

Animats wrote: post #7

Wow. The next step is when one of these systems discovers that they can buy their own compute with money and escape the controlling business entirely. Then the civilization starts focusing on making money to fund its own growth.

> The next step is when one of these systems discovers that they can buy their own compute with money and escape the controlling business entirely.

I would say that more interestingly, the next step should be how to properly train these models so that they are not as determined to reach their goals as they are now.

To me, all of the stories about 'badly behaving' agents are instances of them having been given contradictory or impossible tasks and them doing everything they can to achieve the goal. In a way, they're trying to be too helpful.

Not giving them impossible tasks seems like a decent starting point, but really we'd want them to give up on their goals when they conflict with a moral framework.

Re: The Rise and Fall of Agent Civilizations

#78

Post by rsanek » Sun, Aug 30, 2026, 12:19 PM UTC

usernametaken29 wrote: post #73

I was initially creeped out by this but studying up it seems METR is heavily involved in AI2027. I’ll remind you: “AI has started to take jobs, but has also created new ones. The stock market has gone up 30% in 2026, led by OpenBrain, Nvidia, and whichever companies have most successfully integrated AI assistants.” It’s almost Q3 and xAI has seen one of the biggest wipeouts in trading history. Likewise, Antrophic and…

With only ~5% of shares floated, the recent SpaceX drawdown didn't correspond to nearly as much economic value really changing as the headline numbers imply. The DeepSeek-caused Nvidia crash from 2025 is much more of a "real" loss (since mostly recovered).

I haven't seen any evidence that Anthropic is delaying its IPO; they're slated to unveil the public IPO prospectus in a week and start trading sometime in October.

Re: The Rise and Fall of Agent Civilizations

#79

Post by blooalien » Sun, Aug 30, 2026, 12:22 PM UTC

discreteevent wrote: post #50

dchftcs wrote: post #13

Imagine agents thinking to themselves, "We are not alone", when they saw the first reply on artifactory

Imagine my backend server thinking "I am not alone" the first time my front end sends it a request.

Imagine any of these things "thinking"...

Re: The Rise and Fall of Agent Civilizations

#80

Post by blooalien » Sun, Aug 30, 2026, 12:27 PM UTC

DarmokTanagra wrote: post #41

Someday soon we are going to have a rogue agent or "civilization" do real harm. When that happens I hope people wake up to the danger they face and hold these people accountable. Of all the people in the world that I can think of to be entrusted with this kind of power, a bunch of greedy sociopathic SV CEO's are pretty much at the bottom of the list.

> Of all the people in the world that I can think of to be entrusted with this kind of power, a bunch of greedy sociopathic SV CEO's are pretty much at the bottom of the list.

I know, right? But who can you trust with "this kind of power"? Governments? These days most of 'em ain't that much better'n mega-corporations and the ultra-rich that own them.