Live data from Hacker News

AI Could Defeat All of Us Combined

cold-takes.com

61–70 of 114 posts

Re: AI Could Defeat All of Us Combined

#61
The fact that so many apparently smart people earnestly worry about this really makes me feel like I'm missing something that should be obvious. I'm not going to claim real "expertise" in this kind of topic, but I'm far from clueless. My undergrad was in applied math, and though I went to grad school for CS, the focus was machine learning. It isn't what I ended up doing professionally, and I'm nowhere near up to date on the latest greatest breakthrough in novel ANN architectures, but I'm at least not clueless. I'm aware of the fundamentals in terms of what can be accomplished via purely statistical models being used to predict things, and it can be impressive, but I'm also aware of how large software systems work, and I just don't see how we're even headed toward something like this.

Forget about GPT-N and DALL-E for a second and look at the NRO's Sentient program. It's the closest thing out there to a known real attempt at making something like Skynet. It's trying to automate the full TCPED (tasking, collection, processing, exploitation, and dissemination) cycle of global geointelligence, and well, it's actually trying to do even more than that, but that is unfortunately classified. Except it definitely hasn't achieved what it is trying to do, and probably won't. My wife happens to be the enterprise test lead for one of the main components of this system, where "enterprise test" means they try to get the next versions with all the latest greatest features of all components working together in a UAT environment where each of the involved agencies signs off before the new capabilities can go live.

It's amusing to see the kinds of things that grind the whole endeavor to a halt. Probably more than anything, it's issues with PKI. Networked components can't even establish a session and talk to each other at all if they don't trust each other, but trust is established out of band. Classified spy satellite control systems don't just trust the default CAs that Mozilla says your browser should trust. Intelligent or not, there is no possible code path by which the software itself can decide it doesn't care and it will trust a CA anyway or ignore an expired cert and continue talking to some downstream component because doing so is critical to its continued ability to accomplish anything other than sending scrambled nonsense packets into the ether. GPT-N is great at generating text, but no amount of getting better at that will ever make it capable of live-patching code running in read-only memory to give it new code paths it wasn't compiled with. That has nothing to do with intelligence. It just isn't possible at all. You have to have the physical ability to move in space and type characters into a workstation connected to a totally separate network that code is developed on, which is airgapped from the network code is run on.

We seem to be pretty far from even attempting to make distributed software systems that can honest to God do much of anything at all without human monitoring and intervention beyond several-minute at most batch jobs like generate a few paragraphs of text. Sure, that's great, but where is the leap from that to figuring out why an entire AS goes black and half your system disappears because of a typo'd BGP update that then needs to be fixed out of band over the telephone because you can no longer use the actual network, let alone controlling surveillance and weapons systems that aren't networked to the systems code is being developed on? What is the pathway by which a hugely scaled-up ANN is able to bypass the required human steps that propagate feedback from runtime to development in order to achieve recursive self-improvement? Because that is what it would take to gain control of military systems rather than someone's website by purely automated means, and I don't see how it's even the same class of problem. It isn't a research project any AI team is even working on, I have no idea how you would approach it, but it's the kind of nitty-gritty detail you'd have to actually solve to build an automated world conquering system.

It seems like the answer tends to just be "well, this thing will be smarter than any human, so it'll figure it out." That isn't a very satisfying answer, especially when I'm reasonably sure the person saying it has absolutely no idea how security measures and the resulting operational challenges of automating military command and control systems even work.

Re: AI Could Defeat All of Us Combined

#62
The first main point of this article is already wayyyy off the logical rails.

1. AIs don't even need superhuman cognitive abilities to defeat us!

2. They could just, make a bunch of copies and work together, but like, way harder and faster than normal humans, man!

3. Oh, wait, oops, that's superhuman cognitive abilities.

Re: AI Could Defeat All of Us Combined

#63

This type of thing used to scare me a lot more. But after the events of the last few years, the latest IPCC climate report, and the fact that AI has fallen on its face repeatedly despite expectations, I'm more convinced that we'll destroy ourselves before AI has the chance to take us out. But now that I think about it, the idea of a super intelligent AI simply waiting for humanity to die off naturally instead of goin…

> This type of thing used to scare me a lot more. But after the events of the last few years, the latest IPCC climate report, and the fact that AI has fallen on its face repeatedly despite expectations, I'm more convinced that we'll destroy ourselves before AI has the chance to take us out.

I don't think we'll destroy ourselves, but I am starting to think it might be a good thing for humanity of technological civilization falls on its face.

I think fears about AGI are overhyped by people who've read way too much sci-fi, but there are a lot of technologies out there or that are being developed that seem like they could be setting us up for a kind of stable totalitarianism that uses automation to implement much tighter control than was ever possible before.

The people in the 90s who hyped computers as tools of liberation will probably be proven to be very badly wrong. Analog technologies were better, since they're more difficult and costly to monitor. IMHO, a real samizdat is impossible when everything's connected to the internet. And the internet has proven to be far easier to block and control than shortwave radio.

Re: AI Could Defeat All of Us Combined

#65
Why could you not just unplug it? Any AI is going to need to interface with the external world. It's probably going to be quite large as well and require lots of power + networking. That leaves it quite vulnerable to having its "supply"-lines cut.

AI-doomsayers should spend more time engaging with military history and theory.

Re: AI Could Defeat All of Us Combined

#66

I reckon it will end up like in Dune where the thinking machines enslaved humanity not because they were evil but we just got lazy and outsourced the running of things to them because of hedonism (pursuit of pleasure/satisfaction). Now I don't mean mass sex orgies but doing the daily stuff is such a waste of time and boring - bullshit jobs.

Brian's Dune is not canon, it's just a fanfic.

Re: AI Could Defeat All of Us Combined

#68

Why could you not just unplug it? Any AI is going to need to interface with the external world. It's probably going to be quite large as well and require lots of power + networking. That leaves it quite vulnerable to having its "supply"-lines cut. AI-doomsayers should spend more time engaging with military history and theory.

I mean think of a scenario where the AI picks a side on the American political spectrum. If it proves itself and promises to “eliminate the democrats” I could see a lot of the GOP falling over backwards to do anything they could to make sure this thing doesn’t go offline.

Re: AI Could Defeat All of Us Combined

#69
"they might recruit human allies (through manipulation, deception, blackmail/threats, genuine promises along the lines of "We're probably going to end up in charge somehow, and we'll treat you better when we do") ..."

I think even more likely is the scenario whereby potential human allies are faced with two choices: Be on the losing side until you die (side with the humans), or be on the winning side until you die (side with the bots), and enjoy some power along the way.

Re: AI Could Defeat All of Us Combined

#70
post #5

This doesn't mention AGI, which seems to be the prerequisite to this being a possibility. Despite impressive advances in "weak" ai, strong ai is not a simple extension of weak ai, and it's hard to tell if it will arrive within our lifetime.

I don’t know, DeepMind just released a general purpose AI that should technically be able to help with anything. Who knows at what point that becomes indistinguishable from a human. I don’t know how the leap from running a program that has all the right answers to a conscious “being” with motivations happens though.
Post reply on HN