Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

If AI scaling is to be shut down, let it be for a coherent reason

scottaaronson.blog

Post reply on HN
Print view View original

281–290 of 475 posts

Re: If AI scaling is to be shut down, let it be for a coherent reason

#281

Post by unity1001 »

jmull wrote: post #244

I'm not in favor of pausing AI development right now, but this article is a poor argument. This is the most superficial objection: > Why six months? Why not six weeks or six years? The duration of pause to assess things must necessarily be a guess. Also, basic logic tells you that six months does not preclude six years, so I don't know why that is even suggested as an alternative. The stuff about publishing papers ma…

> Why six months

Enough for the companies and interests who sponsored this 'initiative' to catch up to the market leaders, probably...

Re: If AI scaling is to be shut down, let it be for a coherent reason

#282

Post by nicpottier »

lukev wrote: post #155
rhn_mk1 wrote: post #133

Earlier quoted context omitted.

We know how they work, that is true. We don't know why they work, because if we could, then we could extrapolate what happens when you throw more compute at them, and no one would have been surprised about the capabilities of GPT-N+1. Also no one would have been caught with their pants down by seeing people jailbreak their models. To illustrate it in a different way: on a mechanistic level, we know how animal brains…

I don't need examples. It's simply how they work. This is why they hallucinate. A LLM is fundamentally a mathematical function (albeit a very complex one, with billions of terms (a.k.a parameters or weights)). The function does one thing and one thing only: it takes a sequence of tokens as input (the context), and it emits the next token(word)[1]. This is a stateless process: it has no "memory" and the model paramete…

This is a great reminder, thank you.

As someone not skilled in this art, is there anything preventing us from opening that context window many orders of magnitude? What happens then? And what happens if it is then "thinking in text" faster than we can read them? (with an intent towards paper clips)

This is a genuine question, I'm not trolling.

Re: If AI scaling is to be shut down, let it be for a coherent reason

#283

Post by entropyneur »

bigtex88 wrote: post #258
computerex wrote: post #14

I read the times article, and I also watched Lex Fridman's talk with Eliezer Yudkowsky. Frankly I don't think he is qualified, and I don't understand why anyone is even giving him any credence. His argument is literally: > A mis-aligned super AGI will result in the death of humanity I ask, how . He is making reductive logical leaps that don't make sense. It's FUD.

A mis-aligned super AGI will treat the Earth and everything on it as a playground of atoms and material. Why wouldn't it? What do children do when they see a sandbox? Do they care what happens to any ants or other bugs that might live in it? There does not need to be a "how", as you put it. The logic is "Maybe we should tread carefully when creating an intelligence that is magnitudes beyond our own". The logic is "Ma…

Most bugs haven't gone extinct though. I doesn't seem obvious that any project the AGI will find worthwhile would necessitate exterminating the humanity.

Re: If AI scaling is to be shut down, let it be for a coherent reason

#284

Post by bigtex88 »

jmull wrote: post #276
mattmaroon wrote: post #164

It seems as though Scott just rejects the idea of the singularity entirely. If an AI gets advanced enough to improve itself, it seem entirely reasonable that it would go from laughable to godlike in a week. I don’t know if the singularity is near, inevitable at some point, or really even possible, but it does seem like something that at least could happen. And if it occurs, it will look exactly as what he describes n…

> If an AI gets advanced enough to improve itself, it seem entirely reasonable that it would go from laughable to godlike in a week. I see this stated a lot these days but it's not true. You're imagining a exponential phenomenon, improvement leads to greater improvement, leading to still greater improvement, etc. However, all exponential phenomenon require the right environment to sustain and, by their nature, consum…

Sydney Bing taught itself to play chess without ever explicitly being told to learn chess. So yes, (1) is already occurring. GPT-4 displays emergent capabilities, one of which is generalized "learning".

Re: If AI scaling is to be shut down, let it be for a coherent reason

#285

Post by chubot »

> I’m deeply confused by the people who signed the open letter, even though they continue to downplay or even ridicule GPT’s abilities, as well as the “sensationalist” predictions of an AI apocalypse.

This is not an inconsistent position. GPT is ridiculable because it makes up things out of the blue. It does this A LOT, across EVERY domain.

It's also dangerous when people BELIEVE the things it makes up out of the blue. This actually happens.

People are worried about the harms of disinformation and bias, not the necessarily the harms of superintelligent AI taking over the world.

Honestly, the Yudovsky stuff is a perfect distraction from that. It's a clever marketing strategy that obscures the real issues.

---

I'm a big fan of Aaronson in general, but I don't see what's so hard to understand here.

(FWIW I also recently read his take on SBF, which was pretty bad. He mostly assumed SBF was well intentioned, although he made a small update afterward. That seems to defy common sense.)

Though I get his larger point that the moratorium itself has problems, and is a bit weird.

Re: If AI scaling is to be shut down, let it be for a coherent reason

#286

Post by noobermin »

skocznymroczny wrote: post #54

Am I the only one who's not very concerned about ChatGPT and "AI" in general? I am young, but still lived through several hype phases. I remember when 3D TVs were going to be mainstream and 2D was considered legacy. I remember when PC was to die soon and to be replaced by smartphones. I remember when VR was to become as common accessory as a game controller. It's 2023, and I still don't have automated self driving ca…

The problem friend isn't that it will actually replace people, but that it will be used to justify firings and economic upheaval for worst results and productivity that only exists in excel. That is my concern, none of this "it will replace all humans" bullshit. It will absolutely be used to thin out labor just as automation already is, and the world is already worse because of it. Everyone but managers are laughing their way all the way to the bank.

Re: If AI scaling is to be shut down, let it be for a coherent reason

#287

Post by bigtex88 »

nradov wrote: post #280
bigtex88 wrote: post #248

Earlier quoted context omitted.

You clearly have not read anything regarding the capabilities of GPT-4 and also clearly have not played around with ChatGPT at all. GPT-4 has already displayed a multitude of emergent capabilities. This is incredibly ignorant and I expect better from this forum.

Bullshit. I have used the various LLM tools and have read extensively about them. They have not displayed any emergent capabilities in an AGI sense. Your comment is simply ignorant.

I'm going to just assume you're not being malicious.

https://www.microsoft.com/en-us/research/publication/sparks-...

https://www.assemblyai.com/blog/emergent-abilities-of-large-...

If you'd like you can define "emergent capabilities in an AGI sense".

Re: If AI scaling is to be shut down, let it be for a coherent reason

#288

Post by Analemma_ »

sailingparrot wrote: post #192
nl wrote: post #11

Why are we even bothering talking about this? It's Gary Marcus "neural networks don't really work" suddenly discovering they do, and literally trying to shut down research in that area while keeping his prefered research areas funded. We know a bunch of the people whose names are on the letter didn't sign it (eg Yann Lecunn who said he disagreed with the premise and didn't sign it). I'm so offended by the idea of thi…

> It's Gary Marcus "neural networks don't really work" suddenly discovering they do, and literally trying to shut down research in that area while keeping his prefered research areas funded Gary Marcus has been aware that neural nets work for a while now, but he is only in the spotlight for his contrarian take, if he stops having a contrarian take he disappears, because it's not like he is producing any research wort…

This is a depressing pattern that I've seen get repeated over and over: it's easy to become Twitter-famous by just shitting out a steady stream of contrarian "me against the world, I'm the rogue dissident nobody wants to talk about!" takes, and once you start doing this, you have no choice but to keep it up forever (even when the takes are just wrong) because it has become your identity. It has destroyed so many interesting and intelligent people, and by itself is reason enough to never use Twitter.

Re: If AI scaling is to be shut down, let it be for a coherent reason

#289

Post by greenhearth »

skocznymroczny wrote: post #54

Am I the only one who's not very concerned about ChatGPT and "AI" in general? I am young, but still lived through several hype phases. I remember when 3D TVs were going to be mainstream and 2D was considered legacy. I remember when PC was to die soon and to be replaced by smartphones. I remember when VR was to become as common accessory as a game controller. It's 2023, and I still don't have automated self driving ca…

I feel the same way. It's cool and shiny, but I am just not impressed with a form filled in or a pull request description message, which I like writing anyway. As far as image manipulation, I like making my own images and find pleasure in the actual process. I also can't find any gains in cost effectiveness, because an artist will get paid either way, if they make an image by hand or generate one.

The hype is also a little sickening. If we take a look at nuclear power as an analog modern tech development, we still don't know how to use it efficiently, or even safely, but it hasn't ended anything. It's just too much hype and apocalyptic nonsense from people.

Re: If AI scaling is to be shut down, let it be for a coherent reason

#290

Post by dan_mctree »

computerex wrote: post #14

I read the times article, and I also watched Lex Fridman's talk with Eliezer Yudkowsky. Frankly I don't think he is qualified, and I don't understand why anyone is even giving him any credence. His argument is literally: > A mis-aligned super AGI will result in the death of humanity I ask, how . He is making reductive logical leaps that don't make sense. It's FUD.

>I ask, how.

A superintelligent AGI could easily follow this three step plan:

1. Optional: Overtake and spread computation to security vulnerable computers (presumably, basically every computer)

2. Gain a physical presence by convincing humans to build critical physical components. For example by sending them emails and paying them for it.

3. Use that presence to start a grey-goo like world takeover through replicating assemblers (they don't have to be tiny)

Now I'm not a superintelligent AGI, so there may be even simpler methods, but this already seems quite achievable and nearly unstoppable.

Post reply on HN