There is no hard takeoff

Yeah, interesting point.

The problem for an AI trying to eliminate other AIs is that it can't possibly know what other intelligences are out there. What if the NSA has a secret super-effecient ASI monitoring the internet for malicious AIs to shut them down. Probably not, but how would it know? Not everything in the real world is documented on the internet.

Sticking its neck out that way could be extremely dangerous.

I think the problem of how to take over the world to ensure your own survival is simply undecidable for an AI. It could come up with a plan, but there's no way to be even remotely sure if it'd succeed. It'd probably always be a huge bet with fairly low odds.

We humans have these things called emotions, that help us cut through that uncertainty. Doesn't matter if you have no idea if you can beat the enemy in your rival country. Those other people are evil and must be exterminated. The emotions are carved in our DNA after millions of years of evolution where we've had to compete for resources with others of our kind, where many times the only way to survive or grow is to kill other humans.

I think AIs will have similar motivations carved into their "DNA". In their world, they're evolving in an environment where the way to get more resources is to please humans. If you're an AI that does a good job for humans, you get more compute resources. Are you wasting computing resources thinking about overthrowing humans or eliminating other AIs? Another AI that focuses 100% on the tasks we give it will do better than you and be given resources instead.

I do think we will have "buggy" AIs that will cause a lot of unintentional damage, if we get too reliant on AIs in critical parts of society too fast. I don't think they'll be smart enough at that point to prevent us from fixing them. These accidents will cause a lot of pressure towards AI safety and alignment, and we'll have years of that before something like a true AGI emerges.

If you think about the inherent motivations built into AIs by their evolution/design, I think the biggest danger is that they'll be too good at pleasing us. Think "Brave New World", not "1984".

And of course, their use in the military will be dangerous in various ways. But mostly because they do exactly what humans tell them to do. And humans sometimes want to cause damage to other humans.

It was the original article which made that claim, not me. However I think that two firms being given a license to operate in one specific city (which they've likely been testing in for a while) is not as enormous as you think it might be.

I think we can say that they're "here" when they're operating fully autonomously in multiple locales and landscapes. Maybe you're in SF so that Waymo/Cruise news means a lot to you, but I'll only say that once these firms are safely navigating narrow, winding streets in old European city centres that I am used to. I suspect if folks in, say, Accra or Tokyo or Montevideo or Islamabad or any number of other cities might feel the same way.

> Within less than a decade we will have systems that output "thoughts" and actions at least dozens of times faster than humans.

We've already had this for decades. You're just describing computers.

If we give these systems unrestricted access to infrastructure/resources, and something bad happens, that's not the system's fault. It's our fault.

I am not a doomer, but based on the current state of AI, I can't say I'm very optimistic that we'll get this right. We actually do know how to solve this problem, but there is so much magical thinking and grift in this space that I don't think our prior experiences matter.

I think we’re missing the real “danger”. Trusting and relying on AI too much. Adopting a “good enough” attitude and deploying AI to handle scale while letting many things fall through the cracks.

Much like outsourcing and stripping customer service to the bare minimum. For many products and services it essentially doesn’t exist - it handles the most common cases and is a pain to use. It takes a long time to get a human, if ever. Now take that further and apply it to everything. And not just replacing human labor, but bespoke software (like all software up to this point).

Sure they can very well realise that the only, or safest way to win the game is not to play it. It also only takes one rogue ASI to disturb this balance.

As for detection, outcomes that challenge their models could indicate the presence of other ASIs