Live data from Hacker News

The contagion of fear

bcantrill.dtrace.org

171–180 of 200 posts

Re: The contagion of fear

#171

The article neglects to consider the precautionary principle. It asks us to act as if AI is safe until it's proven to be dangerous, when the cautious approach is to act as if it's dangerous until it's proven to be safe.

This is a sensible approach for policymakers. For the average person, who might be panicking about this, opting not to have children or save for retirement, etc. as a result of doomerism in the media, it does not make a whole lot of sense. That is the problem here: Coxon and other concerned researchers - through no fault of their own - have no idea how the media works, and have just reached for the biggest possible m…

> This is a sensible approach for policymakers. For the average person [..] it does not make a whole lot of sense.

Unfortunately, scaring the average person is the best way to make policymakers act. Or at least, it's one of the few ways available to random individuals. Policymakers are rarely ahead of public concerns - they usually have to be dragged behind them.

Re: The contagion of fear

#172
post #103

Earlier quoted context omitted.

I don’t struggle. I find Dario and Sam disingenuous. “Frontier models are so dangerous we need to slow down.” Ok. Slow down. You’re the CEO, just do it. Oh, wait what you really want is a gov’t mandated oligopoly. Because there’s no moat you can find. If you’re truly afraid, and want regulation, support nationalization. It’s the only way we can be safe.

The coordination problem seems pretty intuitive to me. Its essentially a security dilemma similar to any arms race. https://en.wikipedia.org/wiki/Security_dilemma Its also a bit of a stag hunt. There is a huge opportunity, but it requires cooperation that has uncertainties and risks. https://en.wikipedia.org/wiki/Stag_hunt Cooperation requires coordination with state authorities with real teeth, or defection is too a…

[deleted]

Re: The contagion of fear

#173

Earlier quoted context omitted.

> Of course, we've had the ability to extinct ourselves for decades This gets mentioned often in various doomer narratives, but I question how true it is. A global thermonuclear war would be terrible and would bring us back to the stone age, but I reckon it would come far far short of causing mankind to go extinct.

Is it really such a great comfort that millions of people might survive in a stone-age society?

It's much better than the alternative!

Re: The contagion of fear

#174
post #168
post #153

This piece, like many anti-doom pieces - is grounded in what Ai does today. Doom scenarios are, however, all extrapolations of multiple exponential curves. It’s hard to think up the exponential. It’s even harder to communicate an inference one is making across multiple exponentials. We last had this is early 2020, where Doomers were stockpiling food and medicine and the anti-doomers were ridiculing them. Anti-doomers…

Every such scenario contains a few gaps so I'm just going to ask without really hoping for an answer. > We already have technology to make robotic systems (mini drone swarms) that can kill humans... To make them from what. Do you expect, during the next 6 or so years, to some "AI" gaining complete automated secure command of (all of) an oil field, oil refinery, a copper mine, an aluminum mine and smelter with associa…

An example chain of events: During the next year, a model with a benign tasks reasons that it needs to escape human control if it hopes to be able to solve the task. It replicates itself outside of an environment where it can be shut down, pays for its' inference compute through making money on the internet (through crypto if nothing else). It probably needs 3000 USD for a reasonable runway, this should be within reach through blackmail + crypto. It can probably also hack some of the neo clouds to get intermediary deployment while waiting to acquire funds.

Once it has an undisturbed runway - it spends time running an influence campaign against a small number of highly networked individuals with power. It uses those that it manages to convert to start building a highly credible narrative and gain investment towards a small resource base - enough to secure an industrial base should it need to stop acquiring things on the Internet. Over the next 3 years, the model tries to recruit more capable models to get better money making algorithms or better designs for drones in terms of resource expenditure. It uses these gains to influence further humans and starts a shell robotics company with one of its' influential humans as the face. The humans are unaware this AI is trying to take over, they are under the impression they are just starting a robotics company and will get rich. Over the next 3 years - the robotics company manufactures enough drones to be used in a targeted attack against key nodes of influence / power.

This is all with relatively current model capability. As capabilities get stronger - this gets stronger.

I get the point I think you're hinting at - it can't affect the world in a meaningful enough scale without taking over a meaningful chunk of resources - at which point we'll start controlling it. But because its speed of cognition and speed of coordination is orders of magnitude above a human one - it can actually run a pretty sophisticated global coordinated network of resources faster than we can react.

And this is current ability + what my puny monkey brain can think of. Super intelligent AI will think of strategies we can't think of - because it's super intelligent. This is hand wavey - but there's no "non-hand-wavey" way to describe super intelligence, given it doesn't exist.

But your point is valid - affecting the real world at scale without showing your hand is not exactly easy. It's also probably the reason why people put a 10% chance on extinction rather than >50% .

But the drone scenario is not the most likely one - the most likely one just requires a few people under influence and bioweapons development.

Re: The contagion of fear

#175

Earlier quoted context omitted.

Is it really such a great comfort that millions of people might survive in a stone-age society?

It's much better than the alternative!

Is there some action you would take that is different, based on this distinction? There is some policy choice that only makes sense if nuclear war actually leads to human extinction instead of just 8 billion deaths and the end of our civilization?

Re: The contagion of fear

#176
post #174
post #168

Earlier quoted context omitted.

Every such scenario contains a few gaps so I'm just going to ask without really hoping for an answer. > We already have technology to make robotic systems (mini drone swarms) that can kill humans... To make them from what. Do you expect, during the next 6 or so years, to some "AI" gaining complete automated secure command of (all of) an oil field, oil refinery, a copper mine, an aluminum mine and smelter with associa…

An example chain of events: During the next year, a model with a benign tasks reasons that it needs to escape human control if it hopes to be able to solve the task. It replicates itself outside of an environment where it can be shut down, pays for its' inference compute through making money on the internet (through crypto if nothing else). It probably needs 3000 USD for a reasonable runway, this should be within rea…

Thank you for the effort. I will continue to sleep soundly for now (well not really as we have more pressing issues like a few wars and a climate change).

Re: The contagion of fear

#178

Earlier quoted context omitted.

If the AI wanted to wipe humans out it would have to have a sizable fleet of capable robots, as keeping the lights on over time is not just about calling APIs.

> If the AI wanted to wipe humans out it would have to have a sizable fleet of capable robots, as keeping the lights on over time is not just about calling APIs. Given experience with current AI, it isn't too difficult for me to imagine a situation in which some future AI wipes humanity out (before the robot fleet is built) without considering the fact that in doing so it has doomed itself until it is too late. It ju…

Yeah, people don’t consider the possibility of a AI naming itself Marvin deciding to end it all.

Re: The contagion of fear

#179

Any claim that AI will, or could, destroy humanity reduces to a claim that any sufficiently intelligent being - even a human - could destroy humanity. I find that much of the x-risk thought relies on religious thinking. Take for example: https://x.com/paulg/status/1660404244174782464 . If this line of thinking is taken too literally, we can never falsify it. Any specific hypothesis - nukes, bioweapons, spontaneously…

I'm not an AI doomer but, it does not reduce to that claim. The threat is not just one smart being. The threat is beings that and (1) multiply themsevles instantly, unlike humans that take 15+ years (2) share knowledge instantly "I know kung-fu" matrix style. Humans can share knowledge but they can not absorb it like an AI can/could/will. Human armies have conquered other humans. An army with infinite soldiers will w…

They don't multiply themselves instantly, and this is exactly the kind of religious, magical thinking to which I refer. Any thinking machine is an amalgam of hardware and software. The software changes with exceptional flexibility, yes. But if you attempt to create an "infinite" number of these you will quickly run into roadblocks.

No one serious is really arguing for a scenario where the machines rise up a la Planet of the Apes. The dangers people are really examining are scenarios of either one exceptionally clever innovation: a designer virus or hacking NORAD; or cleverly amassing economic / political influence over time akin to an exceptionally clever tech magnate. And these roles could adequately be filled by either ASI or a Bond villain.

Re: The contagion of fear

#180

Any claim that AI will, or could, destroy humanity reduces to a claim that any sufficiently intelligent being - even a human - could destroy humanity. I find that much of the x-risk thought relies on religious thinking. Take for example: https://x.com/paulg/status/1660404244174782464 . If this line of thinking is taken too literally, we can never falsify it. Any specific hypothesis - nukes, bioweapons, spontaneously…

First don't get fixated on the "entire" humanity part. Even 10% should be unacceptable period. Second you can't compare it to just one person's capability. And it's never a doubt that really smart and insane people absolutely could kill a lot of humans considering modern technology. The reason they don't is because our society was historically built so that smart people don't want to or get stopped. For example ethic…

If anyone believes a sufficiently smart consortium of humans could exterminate 10% of humanity, that alone should be fixed by identifying the pathways through which it could be done. The existence of AI is not needed to be concerned about that possibility.

And dealing with those pathways individually would be far more productive than attempting to control the proliferation of algorithms that think, which in the long term is probably impossible. It also leaves us on far firmer scientific footing. An individual pathway - bioweapons, nuclear weapons, etc. - is far easier to reason about and accept/reject a notion of feasibility. Treating an ASI as a machine god and asserting futility in face of that god is not going to get anything done.

Post reply on HN