Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

If AI scaling is to be shut down, let it be for a coherent reason

scottaaronson.blog

Post reply on HN
Print view View original

161–170 of 475 posts

Re: If AI scaling is to be shut down, let it be for a coherent reason

#161

Post by Jensson »

rhn_mk1 wrote: post #133
lukev wrote: post #125

Earlier quoted context omitted.

This is exactly the kind of mysticism I'm talking about. In fact we know precisely how LLMs work. The fact that parts of human linguistic concept-space can be encoded in a high dimensional space of floating point numbers, and that a particular sequence of matrix multiplications can leverage that to perform basic reasoning tasks is surprising and interesting and useful . But we know everything about how how it is trai…

We know how they work, that is true. We don't know why they work, because if we could, then we could extrapolate what happens when you throw more compute at them, and no one would have been surprised about the capabilities of GPT-N+1. Also no one would have been caught with their pants down by seeing people jailbreak their models. To illustrate it in a different way: on a mechanistic level, we know how animal brains…

> Also no one would have been caught with their pants down by seeing people jailbreak their models.

Preventing jailbreak in a language model is like preventing a GO AI from drawing a dick with the pieces. You can try, but since the model doesn't have any concept of what you want it to do it is very hard to control that. Doesn't make the model smart, it just means that the model wasn't made to understand dick pictures.

Re: If AI scaling is to be shut down, let it be for a coherent reason

#163

Post by virgildotcodes »

maroonblazer wrote: post #66
tibbon wrote: post #27

Earlier quoted context omitted.

I don’t think it even requires that AI to be sentient or malicious. The humans already are. Given a tool for carnage and hatred, people will use it. How long did it take from us getting the atomic bomb working to use in production? Less than three months. Will Putin or terrorists hold back from using it in terrible ways if they have it available to them?

>I don’t think it even requires that AI to be sentient or malicious. The humans already are. We are and we aren't. I was struck by this line in the OP: >AI is manifestly different from any other technology humans have ever created, because it could become to us as we are to orangutans; As far as I can tell, we humans treat orangutans quite kindly. I.e., on the whole, we don't go around killing them indiscriminately o…

We’ve wiped out over 60% of the orangutan population in the last 16 years. We’re literally burning them alive to replace their habitat with palm oil plantations. [0]

We currently kill more animals on a daily basis than we have at any point in human history, and we are doing this at an accelerating rate as human population increases.

The cruelty we inflict on them in industry for food, clothing, animal testing, and casually as collateral damage in our pursuit of exploiting natural resources or disposing of our waste is unimaginable.

None of this is kindness. There are movements to address these issues but so far they represent the minority of action in this space, and have not come close to eclipsing the negative of our relationship to the rest of life on Earth in our present day.

All this is just to say that we absolutely do not want another being to treat us the way we treat other beings.

As to whether AI poses a genuine risk to us in the short term, I’m unsure. In the OP and EY’s article, there was something about Homo sapiens vs Australopithecus.

If it’s one naked Homo sapiens dropped into the middle of 8 billion Australopithecus I’m not too worried about the Australopithecus.

[0]https://www.cnn.com/2018/02/16/asia/borneo-orangutan-populat...

Re: If AI scaling is to be shut down, let it be for a coherent reason

#164

Post by mattmaroon »

It seems as though Scott just rejects the idea of the singularity entirely. If an AI gets advanced enough to improve itself, it seem entirely reasonable that it would go from laughable to godlike in a week. I don’t know if the singularity is near, inevitable at some point, or really even possible, but it does seem like something that at least could happen. And if it occurs, it will look exactly as what he describes now. One day it’ll seem like a cool new tool that occasionally says something stupid and the next it’ll be 1,000 times smarter than us. It won’t be as we are to orangutans though, it’ll be as we are to rocks.

The six month pause though, I don’t think would be helpful. It is hubris to think we could control such an AI no matter what safeguards we try to add now. And since you couldn’t possibly police all of this activity it just seems silly to think a six month pause would do anything other than give companies that ignore it an advantage.

Re: If AI scaling is to be shut down, let it be for a coherent reason

#165

Post by rhn_mk1 »

lukev wrote: post #155
rhn_mk1 wrote: post #133

Earlier quoted context omitted.

We know how they work, that is true. We don't know why they work, because if we could, then we could extrapolate what happens when you throw more compute at them, and no one would have been surprised about the capabilities of GPT-N+1. Also no one would have been caught with their pants down by seeing people jailbreak their models. To illustrate it in a different way: on a mechanistic level, we know how animal brains…

I don't need examples. It's simply how they work. This is why they hallucinate. A LLM is fundamentally a mathematical function (albeit a very complex one, with billions of terms (a.k.a parameters or weights)). The function does one thing and one thing only: it takes a sequence of tokens as input (the context), and it emits the next token(word)[1]. This is a stateless process: it has no "memory" and the model paramete…

I don't see how this proves that asking the model about its internal state will reveal its inner high level processes in a human-readable way.

Perhaps there's a research paper which would explain it better?

Re: If AI scaling is to be shut down, let it be for a coherent reason

#166

Post by FrustratedMonky »

It is probably old fashioned fear mongering. Even if it isn't the end of the world, many Jobs will be 'impacted'. Jobs probably wont be gone gone, but still change, and change is scary. It is true that the GPTs have done some things so amazing that it is waking people up to an uncertain future. VFX artists are already being laid off, Nvidia just demonstrated tech to do a full VFX film using motion capture on your phone. There are other AI initiatives to do for sequence planning, and mapping out tasks that were done for other areas. Pretty soon there wont be an industry that isn't impacted.

But, no stopping it.

Re: If AI scaling is to be shut down, let it be for a coherent reason

#167

Post by rhn_mk1 »

Jensson wrote: post #161
rhn_mk1 wrote: post #133

Earlier quoted context omitted.

We know how they work, that is true. We don't know why they work, because if we could, then we could extrapolate what happens when you throw more compute at them, and no one would have been surprised about the capabilities of GPT-N+1. Also no one would have been caught with their pants down by seeing people jailbreak their models. To illustrate it in a different way: on a mechanistic level, we know how animal brains…

> Also no one would have been caught with their pants down by seeing people jailbreak their models. Preventing jailbreak in a language model is like preventing a GO AI from drawing a dick with the pieces. You can try, but since the model doesn't have any concept of what you want it to do it is very hard to control that. Doesn't make the model smart, it just means that the model wasn't made to understand dick pictures…

It does not make the model smart, but it demonstrates our inablity to control it despite wanting it. That strongly suggests that it's not fully understood.

Re: If AI scaling is to be shut down, let it be for a coherent reason

#168

Post by mark_l_watson »

pkoird wrote: post #148
mark_l_watson wrote: post #135

For me, this is complex. My first impression is that many of the signers work on older methods than deep learning and LLMs. Sour grapes. Of course, real AGI has its dangers, but as Andrew Ng has said, worrying about AGI taking over the world is like worrying about overcrowding of Mars colonies. Both tech fields are far in the future. The kicker for me though is: we live in an adversarial world, so does it make sense…

Far in the future? Just 6 months ago, people believed that ChatGPT like model would take 10-15 years more. I believe that Andrew himself doesn't really understand how LLMs work. In particular, what is about the increase in their parameters that induces emergence and what exactly is the nature of such Emergence. So yeah, AGI might be far into the future but it might just be tomorrow as well.

You are correct about the exponential rate of progress.

I also admit to being an overly optimistic person, so of course my opinion could be wrong.

Re: If AI scaling is to be shut down, let it be for a coherent reason

#169

Post by ghodith »

maroonblazer wrote: post #80
NumberWangMan wrote: post #58

Earlier quoted context omitted.

I don't think Eliezer Yudkowsky is very good at bridging the gap with other people in conversations, because most people haven't thought about this as much. However, while it's terrifying and I hate it, and I keep trying to convince myself that he's wrong, I believe him. The first super-intelligent AI will be an alien kind of intelligence to us. It will not have any of the built-in physical and emotional responses we…

> humans manage to do all sorts of mean things to one another, and the only reason that we haven't wiped ourselves out is that we need each other, We don't need cats or dogs. Or orangutans. Why haven't we wiped them out? Because over the centuries we've expanded our moral circle, not contracted it. What's preventing us from engineering this same principle into GPT-n?

Responding "just program it not to do that" to alignment problems is akin to responding "just add more transistors" to computing problems.

We wouldn't be discussing it if we thought it were so simple.

Re: If AI scaling is to be shut down, let it be for a coherent reason

#170

Post by hollerith »

Sankozi wrote: post #91
nl wrote: post #75

Earlier quoted context omitted.

> An AI trained to end cancer might just figure out a plan to kill everyone with cancer. An AI trained to reduce the number of people with cancer without killing them might decide to take over the world and forcibly stop people from reproducing, so that eventually all the humans die and there is no cancer -- technically it didn't kill anyone! I don't understand this and other paperclip maximizer type arguments. If a…

> "don't kill everyone" really doesn't seem the hardest problem here. And yet you made a mistake - it should be "don't kill anyone". AI just killed everyone except one person.

You are pointing at "the complexity of wishes": if you have to specify what you want with computer-like precision, then it is easy to make a mistake.

In contrast, the big problem in the field of AI alignment is figuring out how to aim an AI at anything at all. Researchers certainly know how to train AIs and tune them in various ways, but no one knows how to get one reliably to carry out a wish. If miraculously we figure out a way to do that, then we can start worrying about the complexity of wishes.

Some researchers, like Eliezer and his coworkers, have been trying to figure out how to get an AI to carry out a wish for 20 years and although some progress has been made, it is clear to me, and Eliezer believes this, too, that unless AI research is stopped, it is probably not humanly possible to figure it out before AI kills everyone.

Eliezer likes to give the example of a strawberry: no one knows how to aim an AI at the goal of duplicating a strawberry down to the cellular level (but not the atomic level) without killing everyone. The requirement of fidelity down to the cellular level requires the AI to create powerful technology (because humans currently do not know how to achieve the task, so the required knowledge is not readily available, e.g., on the internet). The notkilleveryone requirement requires the AI to care what happens to the people.

Plenty of researcher think they can create an AI that succeeds at the notkilleveryone requirement on the first try (and of course if they were to fail on the first try, they wouldn't get a second try because everyone would be dead) but Eliezer and his coworkers (and lots of other people like me) believe that they're not engaging with the full difficulty of the problem, and we desperately wish we could split the universe in two such that we go into one branch (one future) whereas the people who are rushing to make AI more powerful go into the other.

Post reply on HN