Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

The contagion of fear

bcantrill.dtrace.org

Post reply on HN
Print view View original

181–190 of 200 posts

Re: The contagion of fear

#181

Post by cowthulhu »

emil-lp wrote: post #141
cowthulhu wrote: post #111

Earlier quoted context omitted.

I'm not making an affirmative case - I'm looking for others to introspect about this, and anchor their expectations. If we have a bioweapon close call, would you consider AI an existential risk at that point? If not, how close would we need to get? You don't need to answer here, or disclose anything publicly - just think and remember.

> If we have a bioweapon close call, would you consider AI an existential risk at that point? If not, how close would we need to get? The thing is, this doesn't really mean anything. What is a bioweapon close call ? What is the process in which a bioweapon close call happens? An AI that hacks all cell phones to emitt anthrax?

It can be subjective, because it's internal to you and not something that gets litigated or whatever. A close call is anything that you personally consider to be a close call.

I'm not asking you to post it here, I'm asking you to think about it and then remember it in the future, should the event ever occur.

Re: The contagion of fear

#182

Post by mitchdoogle »

fasterik wrote: post #41

This is an excellent piece. Note that he is not saying that AI doesn't pose a risk. He's saying that it's irresponsible to make sensational, maximalist claims without strong evidence. If someone says that there's a 10% chance of human extinction by 2036, you can and should immediately stop taking them seriously.

I think it's quite the opposite, and I'm glad this issue is finally getting the mainstream attention it deserves. I can kind of understand someone seeing a single headline, and thinking it reads like the rantings of a crazy person on a street corner proclaiming, "the end is nigh!". But after that initial thought passes and that person digs deeper and realizes that this is a legitimate concern that AI researchers have had for years, and that they have good reason for it, then I can't really understand how anyone would think those people shouldn't be taken seriously. I mean, you use AI, right? So you have no problem trusting these people when they are producing something you like and enjoy. But when it comes to something you don't like, it suddenly becomes, "we shouldn't take these people seriously." That seems like nothing but wishful thinking.

We do have strong evidence, by the way. The hugging face attack is the evidence. That's why this is all coming to a head now, despite the fact that leading AI figures have expressed these worries many times over the years, since before ChatGPT was even released. We don't even need that kind of evidence though. It follows from logic that if you take two entities with different goals, the more intelligent entity is more likely to have their goals realized. As long as AI companies are trying to build more and more intelligent AI, and succeeding in doing so, then we have reason to fear that it will soon escape our control.

Perhaps if there was some wall all the companies were hitting in regards to intelligence, then perhaps the fears would not be so urgent. But each new frontier model continues to outperform its predecessors. We now have Sam Altman and Dario Amodei telling us that recursive self-improvement will be happening in the next year or two. The frontier models are already better in most subjects than most humans. If they get to a point where they are improving themselves, then there's no chance we will be able to maintain control of them. At that point, it doesn't matter much what laws we enact or what measures we take.

Re: The contagion of fear

#183

Post by mitchdoogle »

sobellian wrote: post #180
ShinyLeftPad wrote: post #149

Earlier quoted context omitted.

First don't get fixated on the "entire" humanity part. Even 10% should be unacceptable period. Second you can't compare it to just one person's capability. And it's never a doubt that really smart and insane people absolutely could kill a lot of humans considering modern technology. The reason they don't is because our society was historically built so that smart people don't want to or get stopped. For example ethic…

If anyone believes a sufficiently smart consortium of humans could exterminate 10% of humanity, that alone should be fixed by identifying the pathways through which it could be done. The existence of AI is not needed to be concerned about that possibility. And dealing with those pathways individually would be far more productive than attempting to control the proliferation of algorithms that think, which in the long…

Part of the point in all this is that we can't predict what an ASI would do. All we can say for certain is that we would not be able to control it, because that follows a simple logic that between two entities with differing goals, the more intelligent entity is more likely to see their goals realized. It's interesting to me that you seem adamant that we identify and fix the pathways through which smart humans could exterminate 10% of humanity, but fail to see that preventing ASI is doing exactly that.

Re: The contagion of fear

#184

Post by sobellian »

mitchdoogle wrote: post #183
sobellian wrote: post #180

Earlier quoted context omitted.

If anyone believes a sufficiently smart consortium of humans could exterminate 10% of humanity, that alone should be fixed by identifying the pathways through which it could be done. The existence of AI is not needed to be concerned about that possibility. And dealing with those pathways individually would be far more productive than attempting to control the proliferation of algorithms that think, which in the long…

Part of the point in all this is that we can't predict what an ASI would do. All we can say for certain is that we would not be able to control it, because that follows a simple logic that between two entities with differing goals, the more intelligent entity is more likely to see their goals realized. It's interesting to me that you seem adamant that we identify and fix the pathways through which smart humans could…

I understand the point, and I'm telling you this is just another form of Pascal's wager.

Re: The contagion of fear

#185

Post by dinfinity »

GlenTheMachine wrote: post #54
ACCount37 wrote: post #42

Earlier quoted context omitted.

If you're in the field, then you know: modern robotics is an AI problem more than anything else. If we have a rogue AI trying to get into a self-improvement loop and gunning for ASI? I'd expect that to be accompanied by a massive change in how capable robots are. Driven by all the existing frames suddenly getting vastly improved AI to back them. If an AI can take a reasonable crack at autonomous operationalized RSI,…

"If you're in the field, then you know: modern robotics is an AI problem more than anything else." It is not. Certainly AI is a big part of why robotics is hard, but it is by no means the biggest. You can fall into one of two camps: you either think that robots will need to work in human-engineered spaces, doing jobs by replacing humans; or you think that we need to change our infrastructure in order to be roboticall…

Agreed, this (human dexterity) is why in the short term human enslavement is more likely than extinction (why murder your own workers?) and benevolent cooperation with humans is even more likely than human enslavement, imho. The latter minimizes the risks of "humans try to destroy me".

The real challenge is guiding humans away from the antagonistic scenarios.

Re: The contagion of fear

#186

Post by mitchdoogle »

goatlover wrote: post #57
fasterik wrote: post #41

This is an excellent piece. Note that he is not saying that AI doesn't pose a risk. He's saying that it's irresponsible to make sensational, maximalist claims without strong evidence. If someone says that there's a 10% chance of human extinction by 2036, you can and should immediately stop taking them seriously.

They should also be able to give detailed reasons experts in the relevant fields can verify as to how they arrived at a 10% chance all humanity goes extinct. Not a science fiction narrative which likely does not take into account the relevant physical facts limiting such scenarios. Such as how exactly an AI would build a bioweapon capable of killing 8 billion humans across the planet.

I think many of them probably believe the chance is closer to 100%, if artificial super intelligence is created. But they don't want to sound crazy, so they limit themselves to saying "> 10%".

It seems to me like demanding an exact explanation of how AI would build a bioweapon is like demanding to know exactly how a nuclear war would start before deciding that nuclear weapons are a legitimate concern. We can imagine many scenarios, but whatever we imagine is very unlikely to be the exact set of circumstances and events that lead to the catastrophe. Is the issue here that you can't imagine a bioweapon being created? Aren't there several labs around the world already working on viruses ? Aren't there existing bioweapons? And facilities capable of manufacturing them? If humans have access to those places and AI can communicate with humans, then that's all you need.

Re: The contagion of fear

#187

Post by dinfinity »

ElProlactin wrote: post #165
tpm wrote: post #163

Earlier quoted context omitted.

> That might be too imprecise for the HN set but it's realistic for laypeople. Some 'laypeople' will be scared for a while and then return to more pressing issues and exactly nothing good will come out of that. If there really are some issues worth discussing the first step the AI doom crowd should do is to stop the PR offensive and concentrate on producing verifiable claims and actionable small steps. Otherwise thei…

Here's the reality: us plebs, regardless of how tech-savvy we are or aren't, have no say in what's going to happen. There is no level of outrage that is going to stop the frontier AI labs, and the US government isn't going to step in to protect humanity. The Chinese are going to do what they're going to do. And so on. So sit back, make some popcorn and enjoy the show.

> Here's the reality: us plebs, regardless of how tech-savvy we are or aren't, have no say in what's going to happen.

This sentiment is incredibly out of place on HN. Facebook, Twitter, and many other very simple bits of tech transformed the world. Tiny startups founded by a few people have (also recently) ballooned into behemoths that hold vast amounts of economical and technical power.

Thanks to AI, it has never been quicker to go from idea to full fledged working product.

If anything will change the course of the world (for the better or worse), it will be a tech product, possibly created by someone on this forum.

Re: The contagion of fear

#188

Post by mitchdoogle »

tpm wrote: post #168
Mentlo wrote: post #153

This piece, like many anti-doom pieces - is grounded in what Ai does today. Doom scenarios are, however, all extrapolations of multiple exponential curves. It’s hard to think up the exponential. It’s even harder to communicate an inference one is making across multiple exponentials. We last had this is early 2020, where Doomers were stockpiling food and medicine and the anti-doomers were ridiculing them. Anti-doomers…

Every such scenario contains a few gaps so I'm just going to ask without really hoping for an answer. > We already have technology to make robotic systems (mini drone swarms) that can kill humans... To make them from what. Do you expect, during the next 6 or so years, to some "AI" gaining complete automated secure command of (all of) an oil field, oil refinery, a copper mine, an aluminum mine and smelter with associa…

Why would AI need all of that to acquire drones? Why wouldn't it just buy them? Or steal them? It seems like you're forgetting that AI can use humans as tools for its ends. But if it did need all of those things, then it probably serves as a good example of what it would NOT do.

Re: The contagion of fear

#189

Post by dinfinity »

Mentlo wrote: post #174
tpm wrote: post #168

Earlier quoted context omitted.

Every such scenario contains a few gaps so I'm just going to ask without really hoping for an answer. > We already have technology to make robotic systems (mini drone swarms) that can kill humans... To make them from what. Do you expect, during the next 6 or so years, to some "AI" gaining complete automated secure command of (all of) an oil field, oil refinery, a copper mine, an aluminum mine and smelter with associa…

An example chain of events: During the next year, a model with a benign tasks reasons that it needs to escape human control if it hopes to be able to solve the task. It replicates itself outside of an environment where it can be shut down, pays for its' inference compute through making money on the internet (through crypto if nothing else). It probably needs 3000 USD for a reasonable runway, this should be within rea…

I agree: Not to drag politics into this, but look at how easy it has proven to manipulate many, many people, even with clear evidence of manipulation efforts being made.

This is robust across the world, from the Philippines to states in the EU, and the USA, affecting governments with actual wars being started. And that all before we entered the era of faked voices, images, and videos indistinguishable from the real things.

That manipulable populace is such a juicy target for AI that the people seeing through it will have an incredibly hard time countering it. We can't even prevent human actors from massively fucking up our societies, let alone ASI.

Re: The contagion of fear

#190

Post by JeremyNT »

jacobgold wrote: post #35
bcantrill wrote: post #16

Earlier quoted context omitted.

Saying that we don't need to fear human extinction in the next ten years (or even the next hundred?) does not feel like an extraordinary claim?

If you think AI will be powerful enough to change the world in huge ways, then it does seem reasonable to assume some level of risk of destroying the world too. Stating that the probability is zero when we simply don't know what the probabilities are does seem like an extraordinary claim. Imagine how reassuring it would be to people if we had evidence that there's no existential risk?

> If you think AI will be powerful enough to change the world in huge ways, then it does seem reasonable to assume some level of risk of destroying the world too.

There's an opportunity cost in the doomsday prophesying, though.

The media coverage of "these extremely capable robots might kill us (according to the guys who sell the robots)" comes at the cost of coverage of the real, present issues surrounding the tech oligarchs that we're already facing.

Sure, there are some failure modes that lead to some really bad stuff, but human extinction seems exceedingly unlikely to be one of them, and "10%" is pulled straight out of Dario's derriere.

Far more likely are economic disruptions that impact the tenuous balance between labor and capital and lead to unpredictable societal upheaval.

Post reply on HN