Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Is there a canonical source for “the argument for AGI ruin” somewhere

twitter.com

Post reply on HN
Print view View original

71–80 of 81 posts

Re: Is there a canonical source for “the argument for AGI ruin” somewhere

#71

Post by ThrowawayR2 »

Swizec wrote: post #37
Cardinal7167 wrote: post #12

Earlier quoted context omitted.

This is what social media’s real damage is. It turned academic progress into arguable points. Your research degree and countless lab hours is equally valid to my two-second no-research hot take, and if you’re not on board with it then you’re swarmed.

> my two-second no-research hot take This is more like watching an argument between Feynman and Oppenheimer. They look like hot takes on twitter, but these guys have spent plenty of time thinking about it. Yudkowsky is an AI ethics and safety researcher, also founder of LessWrong, an HN favorite – https://en.wikipedia.org/wiki/Eliezer_Yudkowsky And Chalmers is a philosopher focusing on consciousness and cognitive sci…

Yudkowsky and LessWrong was regarded as kooky for ages and, after the Roko's Basilisk affair (https://en.wikipedia.org/wiki/Roko%27s_basilisk and read Yudkowsky's hilariously deranged all-caps response), a laughingstock as well. Suggesting that he's on the same plane as serious researchers is inappropriate.

Re: Is there a canonical source for “the argument for AGI ruin” somewhere

#72

Post by bordercases »

mitthrowaway2 wrote: post #59
bordercases wrote: post #58

Earlier quoted context omitted.

He doesn't want to open himself to criticism from people that might actually do a good job of it.

Did he tell you this? That seems like a mean-spirited accusation.

You would have to deal with the community's pseudointellectualism long enough to identify it as a pattern. Taken in isolation it looks like nothing, but the arc of the rationalist community has been insularity combined with CYA until a crisis comes along which makes them both more guarded with others while appearing to "update" in public discussion, using such a limited notion of belief because they view their core texts and blog posts as a canonical view of philosophy, and covering up degenerate behavior like sexual abuse on a regular basis. The real rationalists just move on from this crazy making. I've had to decondition myself like an ex-cultist which is why I'm reacting this way and I hope others can see how he behaves problematically too.

Re: Is there a canonical source for “the argument for AGI ruin” somewhere

#73

Post by bordercases »

p-e-w wrote: post #61
bordercases wrote: post #54

Earlier quoted context omitted.

An argument that makes all of its relevant assumptions clear and using precise distinctions would move "fairly obvious" and "seems to be so sure" and "sounds naive" into crisp statements about our knowledge. That's why it's desirable. Its use goes beyond simply you being convinced.

In the case of arguments about the dangers of AGI, there is no meaningful distinction between assumptions and conclusions. The points being discussed are so fundamental that they don't lend themselves to being broken down further. If "a being that you don't understand and that is better than you at everything is a potential danger to you" doesn't convince someone, I'm not sure what further elaboration could achieve.…

Are you sure you can't be more specific? Wasn't MIRI studying the math of alignment precisely because one should be more specific about alignment? In your view, were they successful or misguided or what?

Re: Is there a canonical source for “the argument for AGI ruin” somewhere

#74

Post by yeck »

fwlr wrote: post #55

Yes, actually, there is. (I have no idea why Yudkowsky didn’t present it, other than “Twitter isn’t a good platform for this kind of question in the first place, and ever since his Time essay and his appearance on the Fridman podcast, his Twitter timeline has been deluged by disagreeable people, making it an even worse place for this kind of question”.) The canonical source for ~all arguments for AGI ruin is Omohundr…

Thanks. While I don't personally need to be convinced of the existential risks for AGI/ASI, I was also genuinely interested in know what might serve as a good, clear "canonical" argument for why the risk is present. Definitely going to keep this link on standby.

Re: Is there a canonical source for “the argument for AGI ruin” somewhere

#75

Post by mikewarot »

It's easy to come up with one.

---

Part one - GPT4 may have more cognitive power than humans:

If you properly train deep networks, they end up approximating the function you train them on. (Even if you don't know what that function is)

The internal cognitive systems that deep networks use are quite inefficient, or at least the ones we've found so far. The internal systems are alien to us. (Or un-aligned, if you need to use that term)

Thus, if you train a deep net to do a task, there exists a internal mismatch between its self-generated mechanisms to do cognition, and the way a person would do it. (An impedance mismatch, in electrical engineering speak)

This impedance mismatch then requires a much larger amount of cognition, inefficiently used in order for a deep net to approximate the output of humans (as in predicting human text).

Thus GPT-4 is possibly, already cognitively superior to humans, internally. It just has to find a path to impedance matching to outside world for us to believe that is true.

---

Part 2: People are greedy

Upon reading a random comment on Hacker News, a forum for Silicon Valley venture capitalists, and those who wish to become one... a person is inspired to figure out how to do some "impedance matching" to better discover and utilize the internal cognitive mechanisms invented in GPT4 during its training, for profit

A cycle of discover and improvement begins, and eventually it is decided that the AI can improve itself, and is given free reign to run things, because "line go up".

Even if GPT4 isn't smarter than us, profit will drive future versions that are.

All the negative effects of this AI are socialized, and all the positive gains are captured.

---

Part 3 -- The past trends, lead to ruin when accelerated by AI

We've already seen the climate change and other limits to growth caused by humans seeking profit. The metaphorical force "Moloch" is a good descriptor of the cause and effect here.

AI driven by Moloch will lead to a singularity event, outside of human control because we let it happen, because "line go up" and people kept getting richer. Until the finite resources of earth are reached, and the system breaks down.

Re: Is there a canonical source for “the argument for AGI ruin” somewhere

#76

Post by jbay808 »

p-e-w wrote: post #65
jbay808 wrote: post #64

Earlier quoted context omitted.

> The real problem is that we simply cannot hope to predict what an entity that is far superior to any human would do, essentially by definition. In some ways. I can't predict Stockfish's next chess move, or else I'd be at least as good at chess as Stockfish. But even though I can't predict the detailed trajectory, I can predict certain aggregates. Like that those moves are on the path towards checkmate.

That's because you know what Stockfish's goal is. That's not true for a hypothetical AGI.

That's where the idea of "instrumental convergence" comes in. For almost any end goal, there are intermediate goals that are nearly universal, like: not wanting to be turned off. Because being forcibly shut down is not compatible with achieving most goals. It's hard to predict exactly how an AGI would avoid being shut down, but like the Stockfish example, if the AGI is more intelligent than us, it's likely it would find a method that succeeds.

It turns out to be quite a delicate problem to create an AI that is compatible with being shut down by an overseer, and neither strives to avoid being shut down nor strives to shut itself down. Here is a paper by MIRI about this[1], as well as a paper by Deepmind presenting one possible solution[2].

[1] https://intelligence.org/files/Corrigibility.pdf

[2] https://www.deepmind.com/publications/safely-interruptible-a...

Good explainer video by Robert Miles: https://www.youtube.com/watch?v=ZeecOKBus3Q

Re: Is there a canonical source for “the argument for AGI ruin” somewhere

#77

Post by c-cube »

mitthrowaway2 wrote: post #31
Baeocystin wrote: post #28

Not-joking answer: James Cameron. Before him, Stanley Kubrick. Our cultural headspace has been primed to see killer robot AI, so we see killer robot AI. Which is not to downplay AGI risk per se- it's a powerful tool, and powerful anything can be dangerous. But the uniquely foomy paperclip maximizer fear? That's a cultural attractor from the bay area through and through.

Are you sure such movies weren't just written because a plausible premise is more compelling than an implausible one?

So, based on this line of reasoning, do you think zombies are an realistic danger in our near future? A lot of films have certainly been made on the subject.

Re: Is there a canonical source for “the argument for AGI ruin” somewhere

#78

Post by rnk »

rendall wrote: post #38
rnk wrote: post #23

Earlier quoted context omitted.

What's the academically supported description/ argument or whatever for why nuclear weapons are dangerous for the world if they exist? Obviously they are dangerous for the world because we might kill everyone if someone makes a mistake or gets mad and shoots one first. The danger of nukes is more obvious than why AI is dangerous.

Sorry, I don't understand. Are you saying that AI is of the same sort of danger as nukes? "The danger of nukes is more obvious than why AI is dangerous." Succinctly stated. This is why people are asking for canonical arguments for AI Ruin.

First, I was surprised about the idea that there must be an exact description of dangers before someone can investigate the dangers and consider them. So I was raising the issue of nukes as obviously dangerous, ai less so, we can consider the dangers of AI or anything in the abstract.

I do see AI as very dangerous though. My personal belief is we'll build AGI, every country will have some of their own. Now consider the next war US versus China say or Russia, people will be doing everything possible to win. Eventually we'll make AI controlled robots or weapons that don't have human controls.

There's a less dangerous, but still incredibly destructive problem of AI, propaganda, fake information, photos, etc, becoming ever more capable of making influential damaging fake news. The US is already suffering under this without AI, 1/3 of the people believe things that aren't true about the election, science, etc. Multiply that by 100 and a worse problem occurs.

Re: Is there a canonical source for “the argument for AGI ruin” somewhere

#79

Post by rnk »

TZubiri wrote: post #33
rnk wrote: post #23

Earlier quoted context omitted.

What's the academically supported description/ argument or whatever for why nuclear weapons are dangerous for the world if they exist? Obviously they are dangerous for the world because we might kill everyone if someone makes a mistake or gets mad and shoots one first. The danger of nukes is more obvious than why AI is dangerous.

What about the Non Proliferation Treaty?

The nuclear non-proliferation treaty has somewhat limited nukes. But now Iran is on the verge, Israel, India, Pakistan , North Korea and other countries must be pursuing them. North Korea has shown the value of them because nobody can attack them or stop them because they do have some nukes and missiles. Israel and Iran are always attacking each other and regardless of the reason or goodness of either side, I'd pick them as the two most likely countries to start a nuclear war. Maybe even worse than Pakistan versus India.

I'd say we need a new strategy on top of the nuclear non-proliferation treaty.

Re: Is there a canonical source for “the argument for AGI ruin” somewhere

#80

Post by p-e-w »

bordercases wrote: post #73
p-e-w wrote: post #61

Earlier quoted context omitted.

In the case of arguments about the dangers of AGI, there is no meaningful distinction between assumptions and conclusions. The points being discussed are so fundamental that they don't lend themselves to being broken down further. If "a being that you don't understand and that is better than you at everything is a potential danger to you" doesn't convince someone, I'm not sure what further elaboration could achieve.…

Are you sure you can't be more specific? Wasn't MIRI studying the math of alignment precisely because one should be more specific about alignment? In your view, were they successful or misguided or what?

My view is that the entire "friendly AI" movement, and indeed the very idea that it is even possible to align an AGI, is sheer hubris.

Assuming the qualities of AGI are what its proponents claim (namely, intelligence far beyond the upper limit of human intelligence), humans "aligning" such an entity is laughable. We've had spacecraft built by highly intelligent engineers lost because they overlooked a metric/imperial conversion, and similar people believe that they can devise a watertight scheme to effectively contain an adversarial superintelligence?

The smartest humans can barely contain very small aspects of the regular Universe, and the Universe isn't even targeting them like an AGI would.

Post reply on HN