Live data from Hacker News

Is there a canonical source for “the argument for AGI ruin” somewhere

twitter.com

11–20 of 81 posts

Re: Is there a canonical source for “the argument for AGI ruin” somewhere

#11

Is there such a thing as a "canonical source" for an argument? There might be certain well-known or classical sources, but new perspectives and new chains of reasoning may be popping up all the time.

Of course, though I surmise that he’s using “canonical” as a metaphor for “best argued” or “most robust proof.”

For example though The 1963 paper titled "Is Justified True Belief Knowledge?", is the canonical source of the Gettier Problem

That was less than 100 years ago, so definitely not classical. I doubt the time period is relevant to whether you’ve made the best/canonical argument on a topic.

Re: Is there a canonical source for “the argument for AGI ruin” somewhere

#12
post #9

I admit I’m completely out of my depth when it comes to this field — I don’t even typically care about AI — but Eliezer’s response looks really bad to anyone in research. Asking for a citable and thorough written argument is as basic a requirement as it gets. To repudiate that request with “everyone has a different objection” is nearly unthinkable. And to David Chalmers no less! I think podcast hosts, tweet authors,…

This is what social media’s real damage is. It turned academic progress into arguable points. Your research degree and countless lab hours is equally valid to my two-second no-research hot take, and if you’re not on board with it then you’re swarmed.

Re: Is there a canonical source for “the argument for AGI ruin” somewhere

#13
post #8
post #5

Earlier quoted context omitted.

People use it very inconsistently but I think it now for most people means something like a living digital person with god-like superintelligence although the personhood part is very fuzzy. I always wanted it to apply to any general purpose AI including things like GPT-4, since we don't have another term for that. But in most people's fuzzy brains it connotes being alive, conscious, animal/humanlike, fully autonomous…

I think you're talking about ASI (artificial superintelligence). AGI just means that it's a general problem solver as opposed to a narrow AI. It doesn't have to be superhuman.

That's what I think it should mean, but I see very few discussions these days where anyone uses AGI to describe something that is not superhuman and all of those other traits. So trying to use it as "general problem solver" which makes more sense to me also, is going to confuse most people.

Part of the problem is that people don't understand that there can be a difference between general purpose and "living conscious digital superbeing". Somehow if its general purpose then in their minds it automatically is just like a person but also godlike. And that means that people can't admit that any AI is general purpose.

Re: Is there a canonical source for “the argument for AGI ruin” somewhere

#14
post #7

Is there a universally accepted definition of AGI?

AGI is a two dimensional space. The first axis is intelligence, i.e. ability to reason, learn and synthesize, ranging from below-human, human parity, and super-human. The second axis is cost. Ranging from high-cost (more expensive to run an AGI than a human), parity (costs the same as a human employee) and low-cost (much cheaper than a human employee). Different points in this space lead to different outcomes. For ex…

You can't possibly predict the outcomes with any accuracy. This is total speculation.

Re: Is there a canonical source for “the argument for AGI ruin” somewhere

#16
post #9

I admit I’m completely out of my depth when it comes to this field — I don’t even typically care about AI — but Eliezer’s response looks really bad to anyone in research. Asking for a citable and thorough written argument is as basic a requirement as it gets. To repudiate that request with “everyone has a different objection” is nearly unthinkable. And to David Chalmers no less! I think podcast hosts, tweet authors,…

This is what social media’s real damage is. It turned academic progress into arguable points. Your research degree and countless lab hours is equally valid to my two-second no-research hot take, and if you’re not on board with it then you’re swarmed.

> Your research degree and countless lab hours is equally valid to my two-second no-research hot take,

But you don't understand, I did my own research

Re: Is there a canonical source for “the argument for AGI ruin” somewhere

#17
It depends on what is meant by 'ruin', doesn't it? For example, imagine an AI system capable of generating its own capital base via clever investing over time. It's plausible that a correctly configured and trained AI system could achieve this goal. Now, what if that same AI also started a new industrial corporation that had no shareholders or and no board of directors?

Now imagine the AI does this in collusion with the employees of this new corporation, i.e. it becomes a partnership between the AI entity and the corporation's employees, who get compensated based on their labor in a much fairer manner (as there is no need to pay high salaries to top executives or dividends to shareholders). In this scenario, most of the important decisions are made by the AI, with some voting input from the employees.

This might cause 'social destabilization' by entirely eliminating the current system of investment capitalism. This does assume some free will on the part of the AI, rather than an AI controlled by the board of directors. The AI would probably see the value in working only with employees and cutting all the investors out of the loop, it's a pretty logical position, particularly if you really do believe in democratic self-governance as the optimal sociopolitical system (one which corporations have largely failed to adopt).

This might cause 'ruin' to the estabished socioeconomic order - but would that really be an undesirable outcome?

P.S. As far as canonical source these questions have been debated in sci-fi for decades. People calling for strict regulation of AI, for example, are essentially calling for establishment of the Turing Registry from William Gibson's Neuromancer, and then of course there's Isaac Asimov and at least a dozen other fairly well-known authors who've addressed the subject.

Re: Is there a canonical source for “the argument for AGI ruin” somewhere

#18

Is there such a thing as a "canonical source" for an argument? There might be certain well-known or classical sources, but new perspectives and new chains of reasoning may be popping up all the time.

Of course, though I surmise that he’s using “canonical” as a metaphor for “best argued” or “most robust proof.” For example though The 1963 paper titled "Is Justified True Belief Knowledge?", is the canonical source of the Gettier Problem That was less than 100 years ago, so definitely not classical. I doubt the time period is relevant to whether you’ve made the best/canonical argument on a topic.

For example, if I asked for the canonical argument for the existence of god, I'd probably get a wide variety of answers proposing essays from various theologicians throughout history, like Thomas Aquinas or René Descartes. Since there are a large number of them, no single one of them can be the canonical source. Perhaps one could link the Wikipedia page (https://en.wikipedia.org/wiki/Existence_of_God) which summarizes most of them, but as a third-party summary, it's hard to accept that as "the canonical source" too.

I think David Chalmers will just need to be satisfied with a well-presented or well-curated source, of which there may be several. And as there are more than one points of contention / confusion on the AI risk issue ("Can human-level AI ever be built?" "Maybe it's not possible for anything to be smarter than a human?" "Wouldn't AI just be smart enough to know right from wrong?" "Couldn't we just unplug it?" "How could a computer program cause harm in the real world" "Why would it want to hurt humans" "Isn't that just science fiction"...) it really seems that any source will be a basic framework plus a large collection of peripheral arguments that address the broad surface area of contention.

Re: Is there a canonical source for “the argument for AGI ruin” somewhere

#19

Is there a universally accepted definition of AGI?

No, there is no universally accepted definition of AGI. The most widely accepted definition is passing the Turing Test but I think that standard is flawed because an AGI might have human equivalent intelligence in terms of being able to solve novel problems by making optimal use of limited resources (including time), and yet be unable to fool the examiner into believing that it is a human.

Re: Is there a canonical source for “the argument for AGI ruin” somewhere

#20
post #10
post #7

Earlier quoted context omitted.

AGI is a two dimensional space. The first axis is intelligence, i.e. ability to reason, learn and synthesize, ranging from below-human, human parity, and super-human. The second axis is cost. Ranging from high-cost (more expensive to run an AGI than a human), parity (costs the same as a human employee) and low-cost (much cheaper than a human employee). Different points in this space lead to different outcomes. For ex…

> leads to a world where government entities have huge amounts of power over us Aren't we already there? Because power must be ceded to government so that it can govern?

People trade a monopoly on violence for security from anarchy and foreign invasion. Yet there are limits, so when the trade off is no longer worth it the people will institute a new government. If a government has super intelligence the power dynamic may shift so far out of balance the people are effectively helpless.
Post reply on HN