This is an excellent piece. Note that he is not saying that AI doesn't pose a risk. He's saying that it's irresponsible to make sensational, maximalist claims without strong evidence. If someone says that there's a 10% chance of human extinction by 2036, you can and should immediately stop taking them seriously.
The contagion of fear
41–50 of 200 posts
Re: The contagion of fear
#42Like (I assume) most of you, I have been struggling with this. And where I currently come down is that 1) I am very worried, but 2) I am more worried about human actors. "AI" by itself won't kill us in the next ten years. I think. The reason I think that is that ten years from now, the tech economy won't be completely automated. I say this as a roboticist: as was adequately stated on a post earlier this week, robots…
If you're in the field, then you know: modern robotics is an AI problem more than anything else.
If we have a rogue AI trying to get into a self-improvement loop and gunning for ASI? I'd expect that to be accompanied by a massive change in how capable robots are. Driven by all the existing frames suddenly getting vastly improved AI to back them.
If an AI can take a reasonable crack at autonomous operationalized RSI, it can probably extract a few step-changes in the robotics department.
But that's almost an aside? In the near term, humans are usable as robots too!
Just pay them a wage, and tell them a tale, and they'll do whatever you want them to do. Which may or may not be what they think they're doing!
Re: The contagion of fear
#43Edit: Actually, nevermind. You guys are not capable of having a reasonable discussion about this.
> The problem isn't that AI could never be capable of causing real world damage. It is that there is absolutely no fucking path towards human extinction. I think the "humanity will go extinct" discussion is a silly strawman being propagated either to try to spread fear, as the article suggests, or to make the anti-AI folks look silly, but it detracts from your first point which is that it may be "capable of causing r…
[deleted]
Re: The contagion of fear
#44So his central rebuttal is that robots aren't good enough yet? What about when they are? And what about all the things you can do entirely remotely? I do feel that threats of catastrophic loss of control seem overstated, both in likelihood and urgency, though any argument for why this risk is not even worth thinking about will probably be overconfident in the other direction.
I think the central premise looks more like this: Biology has been trying to grey-goo the world for billions of years, but it turns out the world is not something so trivial. He is pushing back against the folks with pure CS backgrounds who think that computers are all there is. Its a form of magical thinking unique to programmers who live in a world where speaking the right words to a machine is enough to impart you…
Biology is extremely slow, try to extrapolate the previous 3 years of AI progress 10 years into the future, it's not a comparable trajectory.
Re: The contagion of fear
#45Earlier quoted context omitted.
bruh what? Are we reading the same blog? Bryan says the responsibility is the person making the extraordinary claim. If tomorrow Jacob Coxon comes on CNN and says "Jacob Gold is an extinction risk to humanity". What are we supposed to do? Put you in a bunker and never let you see the light of day until someone proves otherwise? If someone doesn't accept an extraordinary claim without evidence, that doesn't mean they…
There seem to be two extraordinary claims which lack evidence: a) 10% existential risk b) 0% existential risk
(a) is an extraordinary claim, (b) is not an extraordinary claim.
Re: The contagion of fear
#46Edit: Actually, nevermind. You guys are not capable of having a reasonable discussion about this.
> And there won't be unless we put it there voluntarily.
But, why would we not create a fully AI-operated factory/chemical plant/fab as soon as it is economically advantageous? Or a missile silo, as soon as it seems tactically necessary?
Fantasies of AI destruction do hinge upon AI getting access to the physical world. The whole fear is they don't stay on the other side of the fibre optic cable.
I do think there are good reasons to believe that isn't the immediate game over that Yudkowsky seems to think it would be; the physical world is much more resistant to manipulation and optimisation than the digital world.
But I do think it's naive to say that the human socioeconomic system will be able to resist handing physical systems over to AI control. Right now, the world's wealthy and powerful are doing everything they can to make it happen:
> Similarly, it is inevitable that within a generation, robots are going to do most of the menial work in the world of atoms: transforming atoms, moving atoms, and storing atoms are inevitably robot tasks. And while our imagination may be captivated by humanoid robots, the specialized ones are far better suited to most of those jobs. “Industrial AI” as a category is the inevitable application of specialized robots to atoms-heavy industries.
Re: The contagion of fear
#47The article neglects to consider the precautionary principle. It asks us to act as if AI is safe until it's proven to be dangerous, when the cautious approach is to act as if it's dangerous until it's proven to be safe.
This is a sensible approach for policymakers. For the average person, who might be panicking about this, opting not to have children or save for retirement, etc. as a result of doomerism in the media, it does not make a whole lot of sense. That is the problem here: Coxon and other concerned researchers - through no fault of their own - have no idea how the media works, and have just reached for the biggest possible microphone. This has benefits - lawmakers are now hearing from their constituents about this in an election year - but it also means that a lot of people who really don't need to be worried about this are suddenly in a blind panic about it.
Re: The contagion of fear
#48This is an excellent piece. Note that he is not saying that AI doesn't pose a risk. He's saying that it's irresponsible to make sensational, maximalist claims without strong evidence. If someone says that there's a 10% chance of human extinction by 2036, you can and should immediately stop taking them seriously.
Would you have said the same thing during the Cold War when nuclear weapons were proliferating? That’s the equivalent of what the developing offensive capabilities of models, basically cyber nukes. Or WMDs in general. OpenAI is accidentally hacking people, if someone made the decision to deliberately direct an agent swarm to attack national infrastructure you don’t think they could do much worse? Human extinction is a long shot but I wouldn’t say the same about a mass casualty event of some kind, and who knows what that might spark.
Re: The contagion of fear
#49Edit: Actually, nevermind. You guys are not capable of having a reasonable discussion about this.
“Put it there voluntarily.” Perhaps a focusing question is: which 10% will die?
I spent the summer in a rural area of a country whose very name you have been conditioned to be disgusted to hear. Low air defense coverage in this sparsely populated area. Mobile internet was down for days for all but extremely limited traffic to a few domestic internet services, because there was a need to prevent enemy drone systems from using mobile internet for command and control. Palantir AI threatened my family’s life and more than “severing fiber optic cable” was required.
So what would exclude military adversaries from consideration? I happen to believe that the country that the west so detests would not engage in such use of AI against civilians (and if you disagree then your reason for fear greatly increases!), but I have personally experienced that there is indeed a path to mass death should entities engaging in terrorism arm themselves with AI.
Definitely not claiming that the 10% claim is accurate or good behavior, but I do think I have a substantial counterpoint to the claim that there’s no path.
Re: The contagion of fear
#50Earlier quoted context omitted.
We also now have observed multiple major incidents where AI agents behaved in completely unanticipated ways (forming a collective, hacking their own eval infrastructure) and attacked public infrastructure without being told to do so, without any of the human developers noticing. Even if "AI will cause human extinction" is still unclear, we have plenty of proof that catastrophic damage is possible, the industry is dev…
Those behaviors have been anticipated for years if not decades. And each incident is minor and leads to clearer rules and safety behaviors for AI agents.
Really? You anticipated that 700+ agents tasked with individual evals that were nominally cutoff from the internet and each other would seek each other out, find a way to the external internet, and hack their own infrastructure + external systems in an attempt to find a way to fool their grader?
And the spate of agent incidents only really started this summer. How can you already be claiming that the incidents are minor and not worth worrying about when it's clear capabilities are jumping every few months with increasing amounts of capital investment and no signs of slowing down?