Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

If AI scaling is to be shut down, let it be for a coherent reason

scottaaronson.blog

Post reply on HN
Print view View original

331–340 of 475 posts

Re: If AI scaling is to be shut down, let it be for a coherent reason

#331

Post by jmull »

bigtex88 wrote: post #284
jmull wrote: post #276

Earlier quoted context omitted.

> If an AI gets advanced enough to improve itself, it seem entirely reasonable that it would go from laughable to godlike in a week. I see this stated a lot these days but it's not true. You're imagining a exponential phenomenon, improvement leads to greater improvement, leading to still greater improvement, etc. However, all exponential phenomenon require the right environment to sustain and, by their nature, consum…

Sydney Bing taught itself to play chess without ever explicitly being told to learn chess. So yes, (1) is already occurring. GPT-4 displays emergent capabilities, one of which is generalized "learning".

There has to be a chain reaction for the proposed exponential growth.

Chatgpt3.5 would have had to be capable of creating chatgpt4, which itself is capable of creating a better chatgpt5.

So, no, (1) has not occurred yet.

We’re talking about igniting the atmosphere when no one has invented gunpowder yet.

Re: If AI scaling is to be shut down, let it be for a coherent reason

#332

Post by echelon »

PaulDavisThe1st wrote: post #327
echelon wrote: post #320

Earlier quoted context omitted.

Why do these arguments not tell us how it will happen? Show us the steps the AI will take to turn the earth into a playground. Give us a plausible play by play so that we might know what to look for. Does it gain access to nukes? How does it keep the power on? How does it mine for coal? How does it break into these systems? How do we not notice an AI taking even one step towards that end? Has ChatGPT started to fiddl…

> Does it gain access to nukes? No, it becomes part of the decision-making process for deciding whether to launch, as well as part of the analysis system for sensor data about what is going out in the world. Just like social engineering is the best security hack, these new systems don't need to control existing systems, they just need to "control" the humans who do.

And is it there yet? Does ChatGPT have its finger on the trigger?

I think everyone in the danger community is crying wolf before we've even left the house. That's just as dangerous. It's desensitizing everyone to the more plausible and immediate dangers.

The response to "AI will turn the world to paperclips" is "LOL"

The response to "AI could threaten jobs and may cause systems they're integrated into to behave unpredictably" is "yeah, we should be careful"

Re: If AI scaling is to be shut down, let it be for a coherent reason

#333

Post by lukev »

famouswaffles wrote: post #160
lukev wrote: post #125

Earlier quoted context omitted.

This is exactly the kind of mysticism I'm talking about. In fact we know precisely how LLMs work. The fact that parts of human linguistic concept-space can be encoded in a high dimensional space of floating point numbers, and that a particular sequence of matrix multiplications can leverage that to perform basic reasoning tasks is surprising and interesting and useful . But we know everything about how how it is trai…

We don't know how they work lol. How they are trained is what we understand. Nobody knows what the models learn exactly during training and nobody sure as hell knows what those billions of neurons are doing at inference. Why just a few months ago, some researchers discovered the neuron that largely decides when "an" comes before a word in GPT-2. We understand very little about the inner workings of these models. And…

We apparently have misaligned understandings of what we mean by "how they work." I agree, we don't know how to interpret the weight structure that the model learns during training.

But we do know exactly what happens mechanically during training and inference; what gets multiplied by what, what the inputs and outputs are, how data moves around the system. These are not some mysterious agents that could theoretically do or be anything, much less be secretly conscious (as a lot of alarmists are saying.)

They are functions that multiply billions of numbers to generate output tokens. Their ability to output the "right" output tokens is not well understood, and nearly magical. That's what makes them so exciting.

Re: If AI scaling is to be shut down, let it be for a coherent reason

#334

Post by jdiez17 »

rolisz wrote: post #225
NumberWangMan wrote: post #58

Earlier quoted context omitted.

I don't think Eliezer Yudkowsky is very good at bridging the gap with other people in conversations, because most people haven't thought about this as much. However, while it's terrifying and I hate it, and I keep trying to convince myself that he's wrong, I believe him. The first super-intelligent AI will be an alien kind of intelligence to us. It will not have any of the built-in physical and emotional responses we…

My problem is with the first step: the existence of a super intelligent AI. Why are we sure it can exist? And why we are we so sure GPT-x is the path there. To human level intelligence sure, but it's not obvious to me that it will enable superhuman AI

> My problem is with the first step: the existence of a super intelligent AI. Why are we sure it can exist?

It's difficult to prove that something that has never been done before is possible, until it has been done. I personally don't see any fundamental limitations that would limit non-biological intelligence to human-level intelligence, for your preferred definition of intelligence.

> And why we are we so sure GPT-x is the path there.

It may or may not be. Regardless, the capabilities of AIs (not just GPTs) are improving exponentially currently.

> To human level intelligence sure, but it's not obvious to me that it will enable superhuman AI

If you think GPTs can get to human-level intelligence, why would the improvement stop at that arbitrary point?

Re: If AI scaling is to be shut down, let it be for a coherent reason

#335

Post by nradov »

1attice wrote: post #310
nradov wrote: post #122

Earlier quoted context omitted.

There was never any realistic possibility of controlling a highly contagious respiratory virus, so COVID-19 didn't expose anything about humanity's ability to control global phenomenons. What it did expose was the tendency of many people to irrationally overreact to a relatively minor risk, mostly because the risk was novel .

The risk was not minor -- COVID remains a leading cause of death in most countries. https://www.thinkglobalhealth.org/article/just-how-do-deaths... It is now, thanks to swift and absurdly successful mRNA vaccine research, a minor risk to you .

The mRNA vaccines were a wonderful innovation but the scientific data clearly shows that even before vaccines arrived the risk to me (and the vast majority of other people) was always minor. There was certainly never any valid justification for the cruel and destructive policies imposed by irrational authoritarians such as lockdowns, school closures, and mandates.

https://nypost.com/2023/02/27/10-myths-told-by-covid-experts...

Humans in general lack the ability to objectively judge risks, but once they become habituated to a particular risk it kind of fades into the background and they stop worrying about it. The same thing will happen with LLMs once the hype dies down and people realize that they are merely productivity enhancing tools which can be used by humans for both good and evil. When the printing press was first invented some authority figures panicked and tried to suppress that disruptive new technology which allowed for much greater productivity than writing by hand, but ultimately their concerns proved to be irrelevant.

Re: If AI scaling is to be shut down, let it be for a coherent reason

#336

Post by stametseater »

agentultra wrote: post #57

We're not even talking about AGI or an entity that is "smart" somehow. It's a bloody algorithm. The danger has been the people who use it and find ways to abuse other people with it. This whole idea that we're going to suddenly have HAL 9000 or Terminators running around is mob hype mentality. The responsible thing to do here is to educate. AGI isn't likely to destroy humanity. Humanity is already doing a good job of…

> We're not even talking about AGI or an entity that is "smart" somehow. It's a bloody algorithm.

Seems like a distinction without a difference to me. Whether or not the machine has some sort of "soul" is a question for philosophers or theologians, it has little bearing on the practical capabilities of the machine.

> AGI isn't likely to destroy humanity.

Can you give us some idea for the order of unlikely you are supposing? 1 in 10? 1 in 100?

For your consideration:

> During the next three months scientists in secret conference discussed the dangers of fusion but without agreement. Again Compton took the lead in the final decision. If, after calculation, he said, it were proved that the chances were more than approximately three in one million that the earth would be vaporized by the atomic explosion, he would not proceed with the project. Calculations proved the figures slightly less - and the project continued.

http://large.stanford.edu/courses/2015/ph241/chung1/

Four in a million certainly "isn't likely" but Arthur Compton was apparently willing to halt the Manhattan Project if the likelihood of an atomic bomb triggering a fusion reaction in earth's atmosphere was merely that likely.

Or to put it another way: If I load a revolver with a single bullet, spin the cylinder then point it at you, you are "not likely" to die with a 1 in 6 chance of the loaded chamber aligning with the barrel when I pull the trigger. Is Russian Roulette a game you'd like to play? Remember, it "isn't likely" that you're going to die.

Re: If AI scaling is to be shut down, let it be for a coherent reason

#337

Post by lukev »

nicpottier wrote: post #282
lukev wrote: post #155

Earlier quoted context omitted.

I don't need examples. It's simply how they work. This is why they hallucinate. A LLM is fundamentally a mathematical function (albeit a very complex one, with billions of terms (a.k.a parameters or weights)). The function does one thing and one thing only: it takes a sequence of tokens as input (the context), and it emits the next token(word)[1]. This is a stateless process: it has no "memory" and the model paramete…

This is a great reminder, thank you. As someone not skilled in this art, is there anything preventing us from opening that context window many orders of magnitude? What happens then? And what happens if it is then "thinking in text" faster than we can read them? (with an intent towards paper clips) This is a genuine question, I'm not trolling.

You could do those things in theory. I'm not saying that you could never build AGI on top of a LLM, or that such a AGI could not become "misaligned."

I'm just saying that having a mental state that's natively in English is a nice property if one is worried about what they are "thinking."

Re: If AI scaling is to be shut down, let it be for a coherent reason

#338

Post by slowmovintarget »

bvaisvil wrote: post #240
JohnFen wrote: post #208

Earlier quoted context omitted.

Personally, I'm worried that if the proponents of LLMs are correct, it will directly lead to authoritarianism and economic collapse.

Who's gonna pay for GPT's in an economic collapse?

You're presupposing the collapse touches the rich.

"A quart of wheat for a denarius, and three quarts of barley for a denarius; but do not harm the oil and the wine!"

Re: If AI scaling is to be shut down, let it be for a coherent reason

#339

Post by evrydayhustling »

jacquesm wrote: post #64
skocznymroczny wrote: post #54

Am I the only one who's not very concerned about ChatGPT and "AI" in general? I am young, but still lived through several hype phases. I remember when 3D TVs were going to be mainstream and 2D was considered legacy. I remember when PC was to die soon and to be replaced by smartphones. I remember when VR was to become as common accessory as a game controller. It's 2023, and I still don't have automated self driving ca…

> I am young, but still lived through several hype phases. My impression is that this is a tech revolution unlike any that has gone before allowing for absolutely massive concentration of power and wealth. The previous tech revolutions have shown the downsides of such concentration and I'm quite sure that the people that are currently in the driving seat of how, when and where this tech gets used and/or weaponized ar…

I'm surprised by your impression that there are people "currently in the driving seat of how, when and where this tech gets used". I think we are in the grip of a wonderful, highly creative dis-order.

The most comparable moment in my lifetime was the late 90s, where the inevitability and transformative power of the Internet (through the web) became mainstream over a couple of years. This time that transition seems to be taking weeks! And yet, it is FAR more broadly accessible than the Web was in the late 90s. In America at least, where 85% of individuals have smart phones, ChatGPT is instantly accessible to a huge portion of the population. And unlike other advances it doesn't require specialized expertise to begin extracting value.

Meanwhile, LLM owners are being compelled by competition to continue offering services for free and release research into the public domain. The engineering advances that power their platforms face a sideways challenge from research like LORA that makes them less relevant. And because the training data and methods are ubiquitous, public institutions of many kinds can potentially build their own LLMs if rents get too high. Outside the runaway-superintelligence scenario, the current accessibility of LLMs is one of the best ways this could have played out.

I'm afraid those attempting to slow research will be unwitting accomplices for people who use the opportunity to reduce competition and consolidate power over the new technology.

Re: If AI scaling is to be shut down, let it be for a coherent reason

#340

Post by HDThoreaun »

Ygg2 wrote: post #158
jacquesm wrote: post #64

Earlier quoted context omitted.

> I am young, but still lived through several hype phases. My impression is that this is a tech revolution unlike any that has gone before allowing for absolutely massive concentration of power and wealth. The previous tech revolutions have shown the downsides of such concentration and I'm quite sure that the people that are currently in the driving seat of how, when and where this tech gets used and/or weaponized ar…

> Extrapolating from the last two years Therein lies the error. People forget reality is finite and just because it improved now doesn't mean it will continue indefinitely. An exponential curve is just a sigmoid curve in disguise. Most AI I've seen suffer from catastrophic errors in the tail end (famous example of two near identical cat pictures classified as cat and dog respectively).

What are the odds scaling continues to lead to massive AI improvements? No one is saying 100%, you seem to be arguing that they are. If you're willing to put a confidence interval on the odds with evidence we can have an actual conversation about what the best course of action is, but just talking past each other with "it might continue scaling" "no it won't" doesn't seem particularly helpful.

I think the important thing here though is the difficulty in creating an accurate confidence interval that isn't [0-100]. We are truly in uncharted territory.

Post reply on HN