Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Is there a canonical source for “the argument for AGI ruin” somewhere

twitter.com

Post reply on HN
Print view View original

61–70 of 81 posts

Re: Is there a canonical source for “the argument for AGI ruin” somewhere

#61

Post by p-e-w »

bordercases wrote: post #54
p-e-w wrote: post #40

Not sure what kind of "argument" the poster is expecting, but I consider it fairly obvious that an entity that 1. is as far superior to humans as humans are to ants (pick your favorite alternative analogy), and 2. does not share any evolutionary or social commonality with humans is both extremely dangerous and extremely unpredictable. While I'm not completely convinced by the "AGI = annihilation" idea that LessWrong…

An argument that makes all of its relevant assumptions clear and using precise distinctions would move "fairly obvious" and "seems to be so sure" and "sounds naive" into crisp statements about our knowledge. That's why it's desirable. Its use goes beyond simply you being convinced.

In the case of arguments about the dangers of AGI, there is no meaningful distinction between assumptions and conclusions. The points being discussed are so fundamental that they don't lend themselves to being broken down further.

If "a being that you don't understand and that is better than you at everything is a potential danger to you" doesn't convince someone, I'm not sure what further elaboration could achieve. This is a classic problem that arises in many philosophical debates. If there is disagreement about fundamentals, the debate doesn't (and can't) go anywhere.

The real problem is that we simply cannot hope to predict what an entity that is far superior to any human would do, essentially by definition. So I wouldn't frame this as a "debate" so much as two camps of different beliefs. Any "argument" made is like ants trying to understand human ethics. While I consider the aforementioned point to be obvious, I could be wrong about it in ways I can't even comprehend, and so could any other human, regardless of their degree of expertise.

Ironically, this unknown provides something like a meta-argument for extreme caution when dealing with AGI: Just like you wouldn't step blindly into a dark room that might contain a monster, you don't have to know that AGI is dangerous in order to be afraid of it – the mere fact that it might be and you cannot ever know for sure is enough.

Re: Is there a canonical source for “the argument for AGI ruin” somewhere

#63

Post by ChatGTP »

p-e-w wrote: post #40

Not sure what kind of "argument" the poster is expecting, but I consider it fairly obvious that an entity that 1. is as far superior to humans as humans are to ants (pick your favorite alternative analogy), and 2. does not share any evolutionary or social commonality with humans is both extremely dangerous and extremely unpredictable. While I'm not completely convinced by the "AGI = annihilation" idea that LessWrong…

I agree, I also think there is some missing ethical considerations going on too.

If we wanted to create something which is intelligent and potentially becomes sentient (not sure that’s possible but people are trying). We’d have to wear whatever consequences go with it. This may actually come with the desired results of curing cancer, or the undesired results of killing the everything. One thing we’d probably have to admit is, it’s our responsibility or problem to deal with forever after that point. Like when you’re a kid and you wanted a tamagotch, that times infinity.

What if whatever entity we were to create is actually docile and kind of “dumb”, it doesn’t self improve it’s just depressed or something, then what, we just kill it ? They seems wrong. Do we have an obligation to sustain the network supporting it for the rest of time ?

Is alignment actually ethical ? Why should something be enslaved ?

What if an ASI creates a new universe, like a simulated universe, is that then our new purpose ? To sustain the next turtle ?

It’s radical to think about but I agree, there is way way way too much naïveté going jn considering this may be actually possible to achieve sometime soon or down the track. Some more foresight or at least planning would be nice to see, especially from top researchers.

Re: Is there a canonical source for “the argument for AGI ruin” somewhere

#64

Post by jbay808 »

p-e-w wrote: post #61
bordercases wrote: post #54

Earlier quoted context omitted.

An argument that makes all of its relevant assumptions clear and using precise distinctions would move "fairly obvious" and "seems to be so sure" and "sounds naive" into crisp statements about our knowledge. That's why it's desirable. Its use goes beyond simply you being convinced.

In the case of arguments about the dangers of AGI, there is no meaningful distinction between assumptions and conclusions. The points being discussed are so fundamental that they don't lend themselves to being broken down further. If "a being that you don't understand and that is better than you at everything is a potential danger to you" doesn't convince someone, I'm not sure what further elaboration could achieve.…

> The real problem is that we simply cannot hope to predict what an entity that is far superior to any human would do, essentially by definition.

In some ways. I can't predict Stockfish's next chess move, or else I'd be at least as good at chess as Stockfish. But even though I can't predict the detailed trajectory, I can predict certain aggregates. Like that those moves are on the path towards checkmate.

Re: Is there a canonical source for “the argument for AGI ruin” somewhere

#65

Post by p-e-w »

jbay808 wrote: post #64
p-e-w wrote: post #61

Earlier quoted context omitted.

In the case of arguments about the dangers of AGI, there is no meaningful distinction between assumptions and conclusions. The points being discussed are so fundamental that they don't lend themselves to being broken down further. If "a being that you don't understand and that is better than you at everything is a potential danger to you" doesn't convince someone, I'm not sure what further elaboration could achieve.…

> The real problem is that we simply cannot hope to predict what an entity that is far superior to any human would do, essentially by definition. In some ways. I can't predict Stockfish's next chess move, or else I'd be at least as good at chess as Stockfish. But even though I can't predict the detailed trajectory, I can predict certain aggregates. Like that those moves are on the path towards checkmate.

That's because you know what Stockfish's goal is. That's not true for a hypothetical AGI.

Re: Is there a canonical source for “the argument for AGI ruin” somewhere

#66

Post by AstralStorm »

ifdefdebug wrote: post #32
rnk wrote: post #23

Earlier quoted context omitted.

What's the academically supported description/ argument or whatever for why nuclear weapons are dangerous for the world if they exist? Obviously they are dangerous for the world because we might kill everyone if someone makes a mistake or gets mad and shoots one first. The danger of nukes is more obvious than why AI is dangerous.

Well THAT support starts with the experimental records of Manhattan Project and fills a whole library. Support for current ML techniques being dangerous in that sense are handwavy descriptions of what might well be science fiction for another 500 years.

Well, support for ML techniques being stupidly wrong is much bigger. It's the default state.

Then there's some less support for them being misleadingly wrong, much more rarely.

Science fiction Terminators it is not. Paperclip maximizers, however, yes. In fact it can be argued that a typical capitalist company is a variant of a paperclip maximizer and is a simple form of superhuman intelligence.

Nick Bostrom invented that thought experiment. Start from there and go digging through AI safety research of which there are a few journals.

Real life samples include 2010 flash crash, social media automatically optimizing for engagement at any cost, a few computer vision issues causing medical misdiagnosis, Tesla's "self-driving", video game AI exploiting engine bugs rather than playing, suddenly racist face recognition...

And those are just mistakes, a malicious actor can do more with these. DARPA and IARPA have a whole damn wing on handling trojan attacks against ML systems.

A superhuman, even weakly superhuman paperclip maximizer is an extremely dangerous thing. (Analyze climate change in this light...)

Re: Is there a canonical source for “the argument for AGI ruin” somewhere

#67

Post by ilaksh »

ChatGTP wrote: post #50
ilaksh wrote: post #13

Earlier quoted context omitted.

That's what I think it should mean, but I see very few discussions these days where anyone uses AGI to describe something that is not superhuman and all of those other traits. So trying to use it as "general problem solver" which makes more sense to me also, is going to confuse most people. Part of the problem is that people don't understand that there can be a difference between general purpose and "living conscious…

That's what I think it should mean, but I see very few discussions these days where anyone uses AGI to describe something that is not superhuman and all of those other traits. So trying to use it as "general problem solver" which makes more sense to me also, is going to confuse most people. Because almost everyone including top researches assume that once an AGI is a thing, it will quite rapidly become an ASI.

But if you look into detail about what they are saying, they are talking about animal/human-like conscious digital beings that have thousands or millions of times the performance of a human. So they are also making a leap there between something that is general purpose and that.

Re: Is there a canonical source for “the argument for AGI ruin” somewhere

#68

Post by ChatGTP »

ilaksh wrote: post #67
ChatGTP wrote: post #50

Earlier quoted context omitted.

That's what I think it should mean, but I see very few discussions these days where anyone uses AGI to describe something that is not superhuman and all of those other traits. So trying to use it as "general problem solver" which makes more sense to me also, is going to confuse most people. Because almost everyone including top researches assume that once an AGI is a thing, it will quite rapidly become an ASI.

But if you look into detail about what they are saying, they are talking about animal/human-like conscious digital beings that have thousands or millions of times the performance of a human. So they are also making a leap there between something that is general purpose and that.

Yes, that’s why it’s alarming…

Re: Is there a canonical source for “the argument for AGI ruin” somewhere

#69

Post by danaris »

p-e-w wrote: post #49
Cardinal7167 wrote: post #12

Earlier quoted context omitted.

This is what social media’s real damage is. It turned academic progress into arguable points. Your research degree and countless lab hours is equally valid to my two-second no-research hot take, and if you’re not on board with it then you’re swarmed.

> This is what social media’s real damage is. It turned academic progress into arguable points. If that is "damage", the problem must surely lie with academia, not with social media. Academia's only raison d'être is to facilitate formal discourse. Wherever it doesn't do that, it loses its legitimacy. "I'm a researcher and you are not" is not an argument. In fact, I'd go so far as to claim that anyone who makes such a…

I think you may be misunderstanding the GP (though it's possible I am); my reading of that was not that it actually changed what "academic progress" is, but rather that it changed the public perception of it, as it gets discussed publicly on social media, where (particularly on Twitter) it's much more important who can draw the most views/shares/etc than who actually has the better science.

Re: Is there a canonical source for “the argument for AGI ruin” somewhere

#70

Post by nradov »

zarzavat wrote: post #60
nradov wrote: post #14

Earlier quoted context omitted.

You can't possibly predict the outcomes with any accuracy. This is total speculation.

You can say that about any kind of model, can’t you? The point of inventing models is not to predict the future, it’s to better understand the world. The weather app on your phone can’t predict Thursday’s weather. But it can help you know more about Thursday than nothing. You could say that the weather app is “speculating”, indeed it is, so what?

No, you can't say that about any type of model. Weather prediction models forecasting just a few days into the future can be back tested with years of historical data to measure typical accuracy. No such historical data exists for AGI.
Post reply on HN