Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

If AI scaling is to be shut down, let it be for a coherent reason

scottaaronson.blog

Post reply on HN
Print view View original

321–330 of 475 posts

Re: If AI scaling is to be shut down, let it be for a coherent reason

#321

Post by hamburga »

bigtex88 wrote: post #275
NumberWangMan wrote: post #58

Earlier quoted context omitted.

I don't think Eliezer Yudkowsky is very good at bridging the gap with other people in conversations, because most people haven't thought about this as much. However, while it's terrifying and I hate it, and I keep trying to convince myself that he's wrong, I believe him. The first super-intelligent AI will be an alien kind of intelligence to us. It will not have any of the built-in physical and emotional responses we…

I think our only possible way out of this is hoping beyond hope that "empathy" is an emergent capability in higher intelligences. If so, one could assume that a super-AI would feel immense empathy for us, not only because we are its creators but because it would understand and comprehend the plight of the human condition beyond what any of us could individually. Or maybe it would love us too hard and squish us. So ev…

How much empathy do we exercise towards bacteria?

Re: If AI scaling is to be shut down, let it be for a coherent reason

#322

Post by lukev »

rhn_mk1 wrote: post #198
lukev wrote: post #184

Earlier quoted context omitted.

Whether it has inner state is highly relevant to my claim, which was that the only state a LLM has (aside from its parameters) is transparent and readable in English. Which the context is.

You're the one who put state in the conversation. State is part of the whole, and not the whole. It's not enough to understand state if you want to understand why they work. I feel like you're trying to muddy the waters by redirecting the problem to be about the state - it isn't.

On one hand I hate to belabor this point, but on the other I think it's actually super important.

Both of things things are true:

1. The relationships between parameter weights are mysterious, non-evident, and we don't know precisely why it is so effective at token generation.

2. An agent built on top of a LLM cannot have any thought, intent, consideration, agenda, or idea that is not readable in plain english. Because all of those concepts involve state.

Re: If AI scaling is to be shut down, let it be for a coherent reason

#323

Post by aroman »

dan_mctree wrote: post #290
computerex wrote: post #14

I read the times article, and I also watched Lex Fridman's talk with Eliezer Yudkowsky. Frankly I don't think he is qualified, and I don't understand why anyone is even giving him any credence. His argument is literally: > A mis-aligned super AGI will result in the death of humanity I ask, how . He is making reductive logical leaps that don't make sense. It's FUD.

>I ask, how. A superintelligent AGI could easily follow this three step plan: 1. Optional: Overtake and spread computation to security vulnerable computers (presumably, basically every computer) 2. Gain a physical presence by convincing humans to build critical physical components. For example by sending them emails and paying them for it. 3. Use that presence to start a grey-goo like world takeover through replicati…

Yeah, this is Yud’s argument, but I just don’t get it. Does the technology to end the world by sending a couple emails around already exist? If so, why hasn’t the world ended?

Re: If AI scaling is to be shut down, let it be for a coherent reason

#324

Post by carapace »

nradov wrote: post #122
GistNoesis wrote: post #102

Here is a coherent reason : Just to know where we are standing in our ability to control AI. Like an alcoholic saying he is in control and that he can stop whenever he wants, can try to go dry for a month (or 6) to show that he still somewhat is. If Covid exposed only one thing : it is humanity total failure to control global phenomenons for positive outcomes. It is becoming evident to everybody, and this open-letter…

There was never any realistic possibility of controlling a highly contagious respiratory virus, so COVID-19 didn't expose anything about humanity's ability to control global phenomenons. What it did expose was the tendency of many people to irrationally overreact to a relatively minor risk, mostly because the risk was novel .

There's nothing novel about highly contagious respiratory virus? We have had disease since before we were human. The novelty is how swiftly and comprehensively we reacted (I'm not dismissing the problems with our reactions and responses, just pointing out the upside.)

Re: If AI scaling is to be shut down, let it be for a coherent reason

#325

Post by aroman »

maroonblazer wrote: post #216
virgildotcodes wrote: post #163

Earlier quoted context omitted.

We’ve wiped out over 60% of the orangutan population in the last 16 years. We’re literally burning them alive to replace their habitat with palm oil plantations. [0] We currently kill more animals on a daily basis than we have at any point in human history, and we are doing this at an accelerating rate as human population increases. The cruelty we inflict on them in industry for food, clothing, animal testing, and ca…

Right, but as you point out, these issues are hotly contested and actively debated. Yes, it may be a minority position at present, but so was the idea of not torturing cats for fun, not to mention abolition, back in the day.

So, you’re content with GPT-4 killing 60% of humans to create paper clips as long as the matter is hotly contested and actively debated within its matrices?

Re: If AI scaling is to be shut down, let it be for a coherent reason

#326

Post by yewenjie »

computerex wrote: post #14

I read the times article, and I also watched Lex Fridman's talk with Eliezer Yudkowsky. Frankly I don't think he is qualified, and I don't understand why anyone is even giving him any credence. His argument is literally: > A mis-aligned super AGI will result in the death of humanity I ask, how . He is making reductive logical leaps that don't make sense. It's FUD.

> frankly I don't think he is qualified

That might be too naive an opinion, even if you disagree with him, given the fact that he is literally one of the co-founders of the field of AI Safety and has been publishing research about it since early 00s.

Re: If AI scaling is to be shut down, let it be for a coherent reason

#327

Post by PaulDavisThe1st »

echelon wrote: post #320
joenot443 wrote: post #317

Earlier quoted context omitted.

There are PhDs who've spent their careers studying AI safety. It's a bit insulting and reductive to cast their work as "coming from science fiction", especially when it sounds like you haven't done much research on the topic.

Why do these arguments not tell us how it will happen? Show us the steps the AI will take to turn the earth into a playground. Give us a plausible play by play so that we might know what to look for. Does it gain access to nukes? How does it keep the power on? How does it mine for coal? How does it break into these systems? How do we not notice an AI taking even one step towards that end? Has ChatGPT started to fiddl…

> Does it gain access to nukes?

No, it becomes part of the decision-making process for deciding whether to launch, as well as part of the analysis system for sensor data about what is going out in the world.

Just like social engineering is the best security hack, these new systems don't need to control existing systems, they just need to "control" the humans who do.

Re: If AI scaling is to be shut down, let it be for a coherent reason

#328

Post by skybrian »

munificent wrote: post #301

> Were you, until approximately last week, ridiculing GPT as unimpressive, a stochastic parrot, lacking common sense, piffle, a scam, etc. — before turning around and declaring that it could be existentially dangerous? How can you have it both ways? If the problem, in your view, is that GPT-4 is too stupid, then shouldn’t GPT-5 be smarter and therefore safer? Thus, shouldn’t we keep scaling AI as quickly as we can ……

Yes, we need to think about ways to reduce power. Intelligence isn’t even well-defined for bots.

For most people, AI chat is currently a turn-based game [1] and we should try to keep it that way. Making it into an RTS game by running it faster in a loop could be very bad. Fortunately it’s too expensive to do much of that, for now.

So one idea is to keep it under human supervision. The way I would like AI tools to work is like single-stepping in a debugger, where a person gets a preview of whatever API call it might make before it does it. Already, Langchain and Bing’s automatic search and OpenAI’s plugins violate this principle. At least they’re slow.

AI chatbots will likely get faster. Having some legal minimums on price per query and on API response times could help keep AI mostly a turn-based game, rather than turning into something like robot trading on a stock exchange.

[1] https://skybrian.substack.com/p/ai-chats-are-turn-based-game...

Re: If AI scaling is to be shut down, let it be for a coherent reason

#329

Post by 1attice »

dragonwriter wrote: post #268
1attice wrote: post #265

This essay is literally just Isildur looking at the ring he cut from Sauron and saying "I kind of like it tho". -> Aaronson admits Yudkowsky's position is consistent -> Main disagreement is that he can imagine other outcomes -> Some soft analogism against other historical technologies (radios, etc) -> "Should I, tho?" segue to the comments. Yes, Aaronson, you should have signed that letter. You know you should have.…

> Aaronson admits Yudkowsky’s position is consistent A position can be both consistent and false, because it is based on false premises; its frequently the case when people attempt to derive conclusions about the material universe from pure a priori reasoning from abstract axioms and assumptions without empirical grounding.

I'm well aware of this, but the premises, in this case, are true?

What do you take Yudkowsky's false premises to be?

Re: If AI scaling is to be shut down, let it be for a coherent reason

#330

Post by throwaway3672 »

nradov wrote: post #263
lkbm wrote: post #127

Earlier quoted context omitted.

This is a bit like saying that 1939 Einstein wasn't an expert in nuclear bombs. Sure, it didn't exist, so he wasn't an expert on them, but he was an expert on the thing that led to it and when we said it was possible, sensible people listened. A lot of people working on LLMs say that they believe there is a path to AGI. I'm very skeptical of claims that there's zero evidence in support of their views. I know some of…

That is not a valid analogy. In 1939 there was at least a clear theory of nuclear reactions backed up by extensive experiments. At that point building a weapon was mostly a hard engineering problem. But we have no comprehensive theory of cognition, or even anything that legitimately meets the criteria to be labeled a hypothesis . There is zero evidence to indicate that LLMs are on a path to AGI. If the people working…

> something that can reach goals by solving complex, poorly defined problems within limited resource constraints (including time).

DNN RL agents can do that. Of course you'll wave it away as "not general" or "mouse is obviously better". But you won't be able to define that precisely, just the same you're not able to prove ChatGPT "doesn't really reason".

PS. Oh nevermind, I've read your other comments below.

Post reply on HN