Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Is there a canonical source for “the argument for AGI ruin” somewhere

twitter.com

Post reply on HN
Print view View original

21–30 of 81 posts

Re: Is there a canonical source for “the argument for AGI ruin” somewhere

#21

Post by tshaddox »

zug_zug wrote: post #15

Define canonical, but the encyclopedia is a great start: https://en.wikipedia.org/wiki/Existential_risk_from_artifici... Fascinating that even Turing had considered the possibility.

That’s much broader though. It covers the general concept of all potential risks with AI, but I don’t think that’s what Chalmers is asking for. I believe Chalmers is referring specifically to several recent high-profile claims that AI research has a very high likelihood of resulting in the destruction of all humans in the near future and that all AI research should be immediately halted and/or regulated very strictly.

Re: Is there a canonical source for “the argument for AGI ruin” somewhere

#22

Post by tshaddox »

mitthrowaway2 wrote: post #18
AndrewKemendo wrote: post #11

Earlier quoted context omitted.

Of course, though I surmise that he’s using “canonical” as a metaphor for “best argued” or “most robust proof.” For example though The 1963 paper titled "Is Justified True Belief Knowledge?", is the canonical source of the Gettier Problem That was less than 100 years ago, so definitely not classical. I doubt the time period is relevant to whether you’ve made the best/canonical argument on a topic.

For example, if I asked for the canonical argument for the existence of god, I'd probably get a wide variety of answers proposing essays from various theologicians throughout history, like Thomas Aquinas or René Descartes. Since there are a large number of them, no single one of them can be the canonical source. Perhaps one could link the Wikipedia page ( https://en.wikipedia.org/wiki/Existence_of_God ) which summari…

I don’t see that as a problem. If there are numerous mutually exclusive arguments for the same broad conclusion, then feel free to point that out and maybe try to link to one or more of them. It just makes no sense whatsoever that a believer in God would just say “I could never possibly even begin to present an argument because there are so many ways to do so and so many different rebuttals people might make.”

Re: Is there a canonical source for “the argument for AGI ruin” somewhere

#23

Post by rnk »

dvzk wrote: post #9

I admit I’m completely out of my depth when it comes to this field — I don’t even typically care about AI — but Eliezer’s response looks really bad to anyone in research. Asking for a citable and thorough written argument is as basic a requirement as it gets. To repudiate that request with “everyone has a different objection” is nearly unthinkable. And to David Chalmers no less! I think podcast hosts, tweet authors,…

What's the academically supported description/ argument or whatever for why nuclear weapons are dangerous for the world if they exist? Obviously they are dangerous for the world because we might kill everyone if someone makes a mistake or gets mad and shoots one first. The danger of nukes is more obvious than why AI is dangerous.

Re: Is there a canonical source for “the argument for AGI ruin” somewhere

#25

Post by comex »

This link from the Twitter thread is reasonably persuasive (though I disagree with many parts):

https://www.alignmentforum.org/posts/pRkFkzwKZ2zfa3R6H/witho...

Personally, I'm encouraged by the emergence of chain-of-thought prompting for LLMs. Machine learning models have a reputation for being opaque and impossible to interpret. But right now, the best way to get LLMs to perform more complex logical reasoning is to make them write out that reasoning, a mechanism which happens to have built-in interpretability. Perhaps future advances in reasoning will involve more opaque internal states, but it seems plausible to me that the goals of 'be good at human-like reasoning' and 'be able to explain that reasoning (in the way humans do)' will continue to be well-aligned in the future. There would still be the possibility of the AI learning to be deceptive when explaining itself, but it would be much more difficult.

Re: Is there a canonical source for “the argument for AGI ruin” somewhere

#26

Post by mr_toad »

cloudking wrote: post #2

Is there a universally accepted definition of AGI?

And even if there was, could you say for sure that an AI met the definition?

For example self-awareness might be part of a definition of intelligence. But an AI can lie about it’s self awareness.

Re: Is there a canonical source for “the argument for AGI ruin” somewhere

#27

Post by Sharlin »

For an academic book-length treatise on the topic, there’s of course Nick Bostrom’s _Superintelligence_ from 2014. I wonder if Chalmers is aware of it and the basic concepts behind the AGI ruin argument such as orthogonality (a mind as smart as a human does not automagically develop human values) and value convergence (a mind with essentially any goal will derive similar subgoals such as self-preservation, self-improvement, and acquisition of resources).

Re: Is there a canonical source for “the argument for AGI ruin” somewhere

#28

Post by Baeocystin »

Not-joking answer: James Cameron. Before him, Stanley Kubrick. Our cultural headspace has been primed to see killer robot AI, so we see killer robot AI.

Which is not to downplay AGI risk per se- it's a powerful tool, and powerful anything can be dangerous. But the uniquely foomy paperclip maximizer fear? That's a cultural attractor from the bay area through and through.

Re: Is there a canonical source for “the argument for AGI ruin” somewhere

#29

Post by ThrowawayR2 »

rnk wrote: post #23
dvzk wrote: post #9

I admit I’m completely out of my depth when it comes to this field — I don’t even typically care about AI — but Eliezer’s response looks really bad to anyone in research. Asking for a citable and thorough written argument is as basic a requirement as it gets. To repudiate that request with “everyone has a different objection” is nearly unthinkable. And to David Chalmers no less! I think podcast hosts, tweet authors,…

What's the academically supported description/ argument or whatever for why nuclear weapons are dangerous for the world if they exist? Obviously they are dangerous for the world because we might kill everyone if someone makes a mistake or gets mad and shoots one first. The danger of nukes is more obvious than why AI is dangerous.

> "What's the academically supported description/ argument or whatever for why nuclear weapons are dangerous for the world if they exist?"

The Wikipedia article on mutual assured destruction (https://en.wikipedia.org/wiki/Mutual_assured_destruction) traces the origin and evolution of exactly that analysis, including even pre-atom bomb precursors based on earlier weapons that would hypothetically make war "too terrible to ever happen again".

Re: Is there a canonical source for “the argument for AGI ruin” somewhere

#30

Post by mitthrowaway2 »

dvzk wrote: post #9

I admit I’m completely out of my depth when it comes to this field — I don’t even typically care about AI — but Eliezer’s response looks really bad to anyone in research. Asking for a citable and thorough written argument is as basic a requirement as it gets. To repudiate that request with “everyone has a different objection” is nearly unthinkable. And to David Chalmers no less! I think podcast hosts, tweet authors,…

This seems like it's just an issue with the way conversations look on Twitter. If this were two people walking down the street and talking, I don't think that would be taken as a repudiation, just a request to clarify a vague question.

Presumably, Yudkowksy could point to any of several of his own publications, like this one: https://intelligence.org/files/AIPosNegFactor.pdf

Post reply on HN