Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

The contagion of fear

bcantrill.dtrace.org

Post reply on HN
Print view View original

81–90 of 201 posts

Re: The contagion of fear

#81

Post by 0xDEAFBEAD »

GlenTheMachine wrote: post #27

Like (I assume) most of you, I have been struggling with this. And where I currently come down is that 1) I am very worried, but 2) I am more worried about human actors. "AI" by itself won't kill us in the next ten years. I think. The reason I think that is that ten years from now, the tech economy won't be completely automated. I say this as a roboticist: as was adequately stated on a post earlier this week, robots…

I think people are over-focusing on current-day robotics capabilities. First, if we can automate AI research, we can also most likely automate robotics research. But second, I don't even think robots are necessary. See https://slatestarcodex.com/2015/04/07/no-physical-substrate-...

Social engineering tends to be easy by cybersecurity standards. We already had Claude spontaneously attempt social engineering of a malicious pull request on Github in the AISI incident. It was detected, but it easily could've succeeded, and there easily could be malicious AI-requested pull requests which already got accepted that we don't know about. Research suggests that LLMs are pretty good at persuading people.

See also https://aisafety.info/questions/6176/Why-can%E2%80%99t-we-ju...

Re: The contagion of fear

#82

Post by hn_throwaway_99 »

fasterik wrote: post #41

This is an excellent piece. Note that he is not saying that AI doesn't pose a risk. He's saying that it's irresponsible to make sensational, maximalist claims without strong evidence. If someone says that there's a 10% chance of human extinction by 2036, you can and should immediately stop taking them seriously.

Well, I'll have to disagree that this is an excellent piece, but that's another issue. And I do agree that AI killing all humans by 2036 doesn't appear plausible to me. But what is plausible is we could easily be down a path so that by 2036 "future doom" already is a very likely risk.

All of the frontier AI companies have been racing to automate themselves, that is, where AI fully autonomously build the next generation of models. Whether this leads to recursive self improvement is a valid question, but a lot of folks think they are close.

The fear is that a misaligned AI will be building the next model with deliberately hidden motives, similar to some of the behaviors seen in the Hugging Face and related attacks. That is why there is such a big push for interpretability, and why it's highly concerning (a) chains of thought are getting harder to interpret in any case, and (b) companies will go more towards things like looping transformers and "neuralese" where thought processes are completely opaque (i.e. https://www.theinformation.com/articles/secret-technique-beh...)

So the belief is not so much that AI kills us all by 2036, but that instead AI is recursively improving by that time and all seems awesome and great so we put it into more systems that can affect the real world (as we've already begun to do, like literal lethal aerial drones). Things then all go along looking great until AI decides humans are a hindrance to its (hidden) goals.

Again, I think it's fine to argue against specific steps in that scenario, but putting out a blog post saying "this is overhyped bullshit" is not exactly making a cogent argument.

Re: The contagion of fear

#83

Post by thin_carapace »

Jtsummers wrote: post #74
goatlover wrote: post #67

Earlier quoted context omitted.

As bad as it would be, it's unlikely that a full scale nuclear war kills all humans across the planet.

And a full scale nuclear war would require a few more than just one decision. So far, we've never been one decision away from annihilation.

certainly I behave like the article's author and yourself all the time, this isn't meant to be some sort of moral point. watching all these smart people talk so confidently regarding things nobody knows about, it reminds of the confidence ai shows in hallucination. would you say that the singular choice of vasili arkhipov did not prevent annihilation?

Re: The contagion of fear

#84

Post by worik »

jacobgold wrote: post #77
goatlover wrote: post #63

Earlier quoted context omitted.

What's the evidence for there being an existential risk? We are provided scenarios which read like science fiction about RSI and ASI right around the corner, resulting in magic sounding technology that can do anything the person making the claim wants it to do, because it can just make itself smart enough in a short amount of time. But the person making the claim has to actually show how such a thing is possible in t…

The clear and predictable power of AI is the evidence for existential risk. Even if we don't get RSI or ASI, there's a risk we'll automate our world, then it'll break and we'll starve, etc.

> The clear and predictable power of AI is the evidence for existential risk

* The "predictable power of AI" is very advanced predictive text. What a lot you can do with that, and there are clear limits.

* AI has no intent. The greedheads who find themselves in these positions of power have clear intent (often but not always bordering on and actively becoming misanthropic) put their intentions on AI - hence to doom mongering

* Who will starve with the failure of agriculture? A few, a lot, but not everybody. We are good at this - have been doing it a lot longer than computing or science

Re: The contagion of fear

#85

Post by ggm »

Hinton was on the abc radio (Australia) this morning and used far too many unfortunate Anthropomorphisms. He did however acknowledge the unmeasurable theoretical risk of Skymesh was possibly less important than the immediate risk of bad actors.

I find the mental leaps from "in principle could distort BGP based on a closed model of BGP inside the sandbox" to "we meshed an AI into BGP and it instantly distorted global routing and took down all the worlds ambulances and HVAC systems" a bit odd.

Firstly, at least some of the surface of BGP is protected from specious route injections. Secondly, peerings can be dropped and routes blackholed. BGP is under attack from mis-configuration almost constantly. Why is the argument/axiom here that AI is going to instantly corrupt it and "take down the internet" when a large chunk of the Internet (China) is already a virtual island, and runs fine? Does this mean you really wanted to say "Chinese AI will destroy the western Internet" and were too coy about adversarial intent of ... people?

Re: The contagion of fear

#87

Post by fasterik »

hn_throwaway_99 wrote: post #82
fasterik wrote: post #41

This is an excellent piece. Note that he is not saying that AI doesn't pose a risk. He's saying that it's irresponsible to make sensational, maximalist claims without strong evidence. If someone says that there's a 10% chance of human extinction by 2036, you can and should immediately stop taking them seriously.

Well, I'll have to disagree that this is an excellent piece, but that's another issue. And I do agree that AI killing all humans by 2036 doesn't appear plausible to me. But what is plausible is we could easily be down a path so that by 2036 "future doom" already is a very likely risk. All of the frontier AI companies have been racing to automate themselves, that is, where AI fully autonomously build the next generati…

None of that has anything to do with the article, and if you thought the message was "this is overhyped bullshit" then you should go back and read it again. As I already pointed out, he isn't saying anything about the probablity of harms or disasters from AI. He's addressing a specific claim about human extinction, and making a broader point about the responsibility of experts to make measured claims backed up by arguments and evidence.

Re: The contagion of fear

#88

Post by crabmusket »

jchw wrote: post #58
crabmusket wrote: post #46

Earlier quoted context omitted.

> And there won't be unless we put it there voluntarily. But, why would we not create a fully AI-operated factory/chemical plant/fab as soon as it is economically advantageous? Or a missile silo, as soon as it seems tactically necessary? Fantasies of AI destruction do hinge upon AI getting access to the physical world. The whole fear is they don't stay on the other side of the fibre optic cable. I do think there are…

> But, why would we not create a fully AI-operated factory/chemical plant/fab as soon as it is economically advantageous? Or a missile silo, as soon as it seems tactically necessary? Well, for one thing, because it would be dangerous? Why don't you just give Claude Code access to your entire computer without any safeguards? If you wouldn't even give Claude Code unfettered access to your workstation, which really does…

> I am not afraid of AI. I am terrified of the people making decisions, though.

Yes, that is what I was trying to say. Something being obviously dangerous doesn't mean we (edit: they) won't decide to do it anyway.

From a song on an album with a pertinent cover image, "who can stand in the way when there's a dollar to be made?"

Re: The contagion of fear

#89

Post by Jtsummers »

thin_carapace wrote: post #83
Jtsummers wrote: post #74

Earlier quoted context omitted.

And a full scale nuclear war would require a few more than just one decision. So far, we've never been one decision away from annihilation.

certainly I behave like the article's author and yourself all the time, this isn't meant to be some sort of moral point. watching all these smart people talk so confidently regarding things nobody knows about, it reminds of the confidence ai shows in hallucination. would you say that the singular choice of vasili arkhipov did not prevent annihilation?

> would you say that the singular choice of vasili arkhipov did not prevent annihilation?

He prevented a catastrophe, but not annihilation. We were not at risk of that in 1962 and even if he had decided to go with the others, more decisions would have been needed (not just his) to fully escalate to full scale nuclear war. I will reiterate: We have never been one decision away from full scale nuclear war.

But in case you don't understand why, it's because no one person can actually launch all the missiles. And considering the two major arsenals (US and USSR), there has never been a time when two people could make the same decision (launch) and actually launch all the missiles. The orders still have to go out and acted on, many decisions have to be made in order to have full scale nuclear war and come close to annihilation.

Re: The contagion of fear

#90

Post by leoc »

GlenTheMachine wrote: post #69
leoc wrote: post #64

Earlier quoted context omitted.

Right. A hypothetical superhuman AI wouldn’t have to master robotics to affect the physical world. It could simply bribe, blackmail, manipulate and play politics with humans. As others have pointed out, our political leaders have already been playing these games since forever ago https://news.ycombinator.com/item?id=49689978 and a super-AI would be better at it. At the cost of being seen to cite a SF novel in defence…

By definition, if you're taking over the world by bribing humans to be your hands, you aren't killing all the humans. I"m not saying it's obviously going to be great. I'm saying that "extinction event" has a very specific definition, and this isn't it.

It isn't true by definition: you could quite happily induce people to release a series of highly contagious bioweapons, after which those people would be surplus to requirements. What is true is that you're likely to need humans to sustain you and act for you for a few years to decades, so if you're not suicidal or deeply mad (and that is itself by no means self-evident) then total and immediate human extinction is probably not something you will aim for. But ruling out total, prompt human extinction isn't, by itself, remotely enough to justify the OP's overall don't-worry conclusion.

(Again, to be clear, I myself am not predicting or assigning a significant probability to any doom scenarios, because I do not expect AGI.)

Post reply on HN