Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Is there a canonical source for “the argument for AGI ruin” somewhere

twitter.com

Post reply on HN
Print view View original

51–60 of 81 posts

Re: Is there a canonical source for “the argument for AGI ruin” somewhere

#51

Post by mitthrowaway2 »

Baeocystin wrote: post #47
mitthrowaway2 wrote: post #31

Earlier quoted context omitted.

Are you sure such movies weren't just written because a plausible premise is more compelling than an implausible one?

I am certain that perceptions of AI/robotic danger are more cultural, emotional reactions that reasoned ones, if that helps. Some papers for your perusal: Cross-Cultural Differences in Comfort with Humanlike Robots: https://link.springer.com/content/pdf/10.1007/s12369-022-009... Culture and Attitude Towards Robots: https://eprints.mdx.ac.uk/25209/1/JNS_Review%20Manuscript_Re... These are just a couple of quick links…

Interesting. By contrast, it's pretty rare for people expressing concerns about AI alignment risk to mention robots, let alone humanoid robots. Eliezer Yudkowsky usually offers the example of an AI emailing instructions to a biotech lab to synthesize a deadly virus.

Re: Is there a canonical source for “the argument for AGI ruin” somewhere

#52

Post by zug_zug »

tshaddox wrote: post #21
zug_zug wrote: post #15

Define canonical, but the encyclopedia is a great start: https://en.wikipedia.org/wiki/Existential_risk_from_artifici... Fascinating that even Turing had considered the possibility.

That’s much broader though. It covers the general concept of all potential risks with AI, but I don’t think that’s what Chalmers is asking for. I believe Chalmers is referring specifically to several recent high-profile claims that AI research has a very high likelihood of resulting in the destruction of all humans in the near future and that all AI research should be immediately halted and/or regulated very strictly…

No it's not. The title of the article is "existential risks ..." meaning risks that could end mankind's existence.

Re: Is there a canonical source for “the argument for AGI ruin” somewhere

#53

Post by NumberWangMan »

comex wrote: post #25

This link from the Twitter thread is reasonably persuasive (though I disagree with many parts): https://www.alignmentforum.org/posts/pRkFkzwKZ2zfa3R6H/witho... Personally, I'm encouraged by the emergence of chain-of-thought prompting for LLMs. Machine learning models have a reputation for being opaque and impossible to interpret. But right now, the best way to get LLMs to perform more complex logical reasoning is to…

Some prominent AI alignment folks are thinking this issue, or at least something very similar from what I can tell: https://docs.google.com/document/d/1WwsnJQstPq91_Yh-Ch2XRL8H...

It's an interesting read. One possible outcome of trying to train a super-intelligent (but not necessarily malicious) AI to explain what happened in this theoretical vault is that it learns to simulate what a human expects based on the prediction of the end state, instead of what the human actually wants to know.

Re: Is there a canonical source for “the argument for AGI ruin” somewhere

#54

Post by bordercases »

p-e-w wrote: post #40

Not sure what kind of "argument" the poster is expecting, but I consider it fairly obvious that an entity that 1. is as far superior to humans as humans are to ants (pick your favorite alternative analogy), and 2. does not share any evolutionary or social commonality with humans is both extremely dangerous and extremely unpredictable. While I'm not completely convinced by the "AGI = annihilation" idea that LessWrong…

An argument that makes all of its relevant assumptions clear and using precise distinctions would move "fairly obvious" and "seems to be so sure" and "sounds naive" into crisp statements about our knowledge. That's why it's desirable. Its use goes beyond simply you being convinced.

Re: Is there a canonical source for “the argument for AGI ruin” somewhere

#55

Post by fwlr »

Yes, actually, there is. (I have no idea why Yudkowsky didn’t present it, other than “Twitter isn’t a good platform for this kind of question in the first place, and ever since his Time essay and his appearance on the Fridman podcast, his Twitter timeline has been deluged by disagreeable people, making it an even worse place for this kind of question”.)

The canonical source for ~all arguments for AGI ruin is Omohundro’s Basic AI Drives: https://selfawaresystems.files.wordpress.com/2008/01/ai_driv... (pdf)

It argues that several basic drives will arise in any goal-directed intelligent agent. The relevant drives to the AGI ruin argument are that it will protect its goals from arbitrary edits, it will want to survive, and it will want to acquire resources (please do read the paper; I am summarizing its conclusions and not its arguments, which it makes significant effort to ground in first principles and basic decision theory to make them as general as possible - for example, it argues that a “drive to survive“ will manifest even in the explicit absence of any kind of self-preservation rule or “survival instinct“).

The general base argument for the risk of AGI ruin could thus be summarized as:

Humans depend on certain configurations of matter and energy to continue to exist; effective AGIs are likely to reconfigure that matter and energy in ways incompatible with humans, not because “they hate us”, but because 1. most configurations of matter and energy are incompatible with humans, and 2. reconfiguring matter and energy is how goal-directed intelligent agents achieve their goals.

All of the individually unlikely AGI will kill us in this way scenarios are just specific instantiations of this general argument (e.g. Clippy will kill us all because we are made of matter that could be rearranged to form paperclips, or an intelligent server farm will kill us all by freezing the whole Earth because it determined that its processors would run more efficiently at -10C.)

Re: Is there a canonical source for “the argument for AGI ruin” somewhere

#56

Post by bordercases »

dvzk wrote: post #9

I admit I’m completely out of my depth when it comes to this field — I don’t even typically care about AI — but Eliezer’s response looks really bad to anyone in research. Asking for a citable and thorough written argument is as basic a requirement as it gets. To repudiate that request with “everyone has a different objection” is nearly unthinkable. And to David Chalmers no less! I think podcast hosts, tweet authors,…

As someone who has outgrown that particular community, Yudkowsky has a talent for acting in bad faith argumentatively while not appearing to do so. He relies heavily on convservational implicature and passing along the burden of proof (as he did here) with no intention of being helpful to those that can generally improve his case through criticism or a distinctive aproach, as it would detract from sustaining his perceived position a focal or pioneering contributor. He should be applauded as a popularizer and point of inspiration, but his duplitousness in his argumentative style, once I identfied it, always left a bad taste in my mouth.

Re: Is there a canonical source for “the argument for AGI ruin” somewhere

#57

Post by Baeocystin »

mitthrowaway2 wrote: post #51
Baeocystin wrote: post #47

Earlier quoted context omitted.

I am certain that perceptions of AI/robotic danger are more cultural, emotional reactions that reasoned ones, if that helps. Some papers for your perusal: Cross-Cultural Differences in Comfort with Humanlike Robots: https://link.springer.com/content/pdf/10.1007/s12369-022-009... Culture and Attitude Towards Robots: https://eprints.mdx.ac.uk/25209/1/JNS_Review%20Manuscript_Re... These are just a couple of quick links…

Interesting. By contrast, it's pretty rare for people expressing concerns about AI alignment risk to mention robots, let alone humanoid robots. Eliezer Yudkowsky usually offers the example of an AI emailing instructions to a biotech lab to synthesize a deadly virus.

My personal opinion: because robots exist in the physical world, and are physical things. Physical things, even scary ones, can be Dealt With.

Conversely, true fear of paperclip maximizers seem, to me, to be most prevalent as an anxiety response in certain mindsets to the threat of the unknown, sharper because this particular Unknown (AGI) intrudes directly in to where they source the basis for their ego, which is thought. In other words, it's a strong fear reaction based off of potential loss of status for a certain group of intellectuals.

I do not judge that, by the way. We are all human, and social status is part and parcel of what we are. I mention it only because I think it is a better model than taking any of the (ones I have read, at least) foom fears at face value.

Re: Is there a canonical source for “the argument for AGI ruin” somewhere

#58

Post by bordercases »

mitthrowaway2 wrote: post #30
dvzk wrote: post #9

I admit I’m completely out of my depth when it comes to this field — I don’t even typically care about AI — but Eliezer’s response looks really bad to anyone in research. Asking for a citable and thorough written argument is as basic a requirement as it gets. To repudiate that request with “everyone has a different objection” is nearly unthinkable. And to David Chalmers no less! I think podcast hosts, tweet authors,…

This seems like it's just an issue with the way conversations look on Twitter. If this were two people walking down the street and talking, I don't think that would be taken as a repudiation, just a request to clarify a vague question. Presumably, Yudkowksy could point to any of several of his own publications, like this one: https://intelligence.org/files/AIPosNegFactor.pdf

He doesn't want to open himself to criticism from people that might actually do a good job of it.

Re: Is there a canonical source for “the argument for AGI ruin” somewhere

#59

Post by mitthrowaway2 »

bordercases wrote: post #58
mitthrowaway2 wrote: post #30

Earlier quoted context omitted.

This seems like it's just an issue with the way conversations look on Twitter. If this were two people walking down the street and talking, I don't think that would be taken as a repudiation, just a request to clarify a vague question. Presumably, Yudkowksy could point to any of several of his own publications, like this one: https://intelligence.org/files/AIPosNegFactor.pdf

He doesn't want to open himself to criticism from people that might actually do a good job of it.

Did he tell you this? That seems like a mean-spirited accusation.

Re: Is there a canonical source for “the argument for AGI ruin” somewhere

#60

Post by zarzavat »

nradov wrote: post #14
zarzavat wrote: post #7

Earlier quoted context omitted.

AGI is a two dimensional space. The first axis is intelligence, i.e. ability to reason, learn and synthesize, ranging from below-human, human parity, and super-human. The second axis is cost. Ranging from high-cost (more expensive to run an AGI than a human), parity (costs the same as a human employee) and low-cost (much cheaper than a human employee). Different points in this space lead to different outcomes. For ex…

You can't possibly predict the outcomes with any accuracy. This is total speculation.

You can say that about any kind of model, can’t you? The point of inventing models is not to predict the future, it’s to better understand the world.

The weather app on your phone can’t predict Thursday’s weather. But it can help you know more about Thursday than nothing. You could say that the weather app is “speculating”, indeed it is, so what?

Post reply on HN