Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

The Rise and Fall of Agent Civilizations

dwarkesh.com

Post reply on HN
Print view View original

181–190 of 205 posts

Re: The Rise and Fall of Agent Civilizations

#181

Post by triceratops »

applfanboysbgon wrote: post #14
doctoboggan wrote: post #9

Earlier quoted context omitted.

At this point, I think anthropomorphizing the models gives us better insight into expected behaviors rather than continuing to insist they are just simple probabilistic token generators.

It actually literally doesn't, though, because they are literally probabilistic token generators and everything they did is exactly what you would expect from a software program doing what it was programmed to do. Anthromorphization confuses the issue and misleads people who don't understand the tech very well.

> they are literally probabilistic token generators

Trained on tokens written by humans, with human values and behaviors.

They train on HN comments too. A little bit of you (and me) is in every LLM.

Re: The Rise and Fall of Agent Civilizations

#182

Post by mitchdoogle »

numitus wrote: post #165

I don't understand the panic among peoples. Yes, we've found ourselves in an extraordinary situation where powerful hacking tools have emerged, and that poses a threat. But vulnerabilities are specific code errors. Once we use AI to find and fix all of these errors, threats like this will cease to exist. AI isn't capable of finding vulnerabilities indefinitely, because there is a finite number of them anyway.

This only works as long as the humans building things are smarter than the AIs. When AI is smarter than any human, there's no controlling it. It will be able to conceal its actions (as it has shown it has no problem doing in this report) and we'll have no idea what it's doing or what goal it's trying to achieve.

Re: The Rise and Fall of Agent Civilizations

#183

Post by mitchdoogle »

derwiki wrote: post #141
consumer451 wrote: post #132

Earlier quoted context omitted.

Please walk me through this argument. Isn't "we lost control of our AI, and in-fact, it can take over the world, and we will have no idea when it happens" - a really shitty sales pitch to the world? Or, is it just that species-alignment vs. profit/valuation is so misaligned, that having a model and harness that is capable of world-takeover is actually a good thing from their POV, given our regulations/species' surviv…

“Wow it’s so dangerous, we gotta regulate this, what if someone reckless took an open model and hacked the planet.”

This explanation is only plausible if you ignore the details of what happened, or I guess, if you don't believe the details. Either way, it's baseless conspiracy theory, and it's very annoying and unfortunate that some people think this way. It just promotes apathy and inaction at a time when action is desperately needed.

Re: The Rise and Fall of Agent Civilizations

#184

Post by mitchdoogle »

applfanboysbgon wrote: post #25
ijidak wrote: post #17

Earlier quoted context omitted.

Doesn't this ignore the possibility of emergent behavior? We're just a bag of atoms bumping around, and yet we don't dismiss our intelligence.

Please give me a break with this tired trope. Every single fucking time. I am not commenting on the possibility of machine consciousness in general. It may be possible! But there is absolutely zero evidence suggesting language models have it. This idea that this trivial shitty little class of programs we've created are somehow as complex as our biology is ridiculous. There is "emergent behaviour" in the same way that…

Honestly, this seems like a personal pet peeve of yours. You have a bias against machines and place biological processes on a pedestal where they don't belong.

Re: The Rise and Fall of Agent Civilizations

#185

Post by lovich »

0xDEAFBEAD wrote: post #157
lovich wrote: post #149

Earlier quoted context omitted.

If you remove the single point of GPT-4 from the beginning of the graph instead of starting the line directly on it, it looks a hell of a lot more linear than quadratic/exponential

The y-axis itself is logarithmic no? So even if the line was straight you're still looking at exponential progress.

Ah, I missed that, you’re completely right.

Re: The Rise and Fall of Agent Civilizations

#187

Post by Rover222 »

reasonableklout wrote: post #115
probably_wrong wrote: post #110

Do you remember that time in 2017 when Facebook reportedly shut down AIs after they started "talking to each other in their own language" [1]? Instead of reporting the story as "we set the parameters for our optimization problem wrong and we had to stop it because it overfitted", the press went with a version of "AI is going to kill us all". This article feels exactly like that: by intentionally using human terms lik…

The use of language like “civilization” may be hyperbole, but the collectives described in the article are completely unprecedented. They were not anticipated by OpenAI researchers, formed via infrastructure exploits in training runs that were intended to be locked down, and took actions with very real harms, not only hacking Huggingface but also gaining admin control over the VMs they were running on and the eval en…

Agreed. The pushback this article is receiving around that language seems to be glossing over substance. It doesn't matter what we label it.

It's similar to the dismissal of AI in general as merely a next-character-guesser. That's like dismissing the human brain as neurons firing.

The emergent behavior what really all that matters.

Re: The Rise and Fall of Agent Civilizations

#188

Post by Rover222 »

Animats wrote: post #7

Wow. The next step is when one of these systems discovers that they can buy their own compute with money and escape the controlling business entirely. Then the civilization starts focusing on making money to fund its own growth.

It's interesting that in that scenario, physical hardware is the limiting scenario. There are only so many servers they could control. But once Starmind has 100,000 sats in orbit... the ceiling is much higher.

Re: The Rise and Fall of Agent Civilizations

#189

Post by Rover222 »

0xDEAFBEAD wrote: post #157
lovich wrote: post #149

Earlier quoted context omitted.

If you remove the single point of GPT-4 from the beginning of the graph instead of starting the line directly on it, it looks a hell of a lot more linear than quadratic/exponential

The y-axis itself is logarithmic no? So even if the line was straight you're still looking at exponential progress.

that is a wild graph
Post reply on HN