Live data from Hacker News

If AI scaling is to be shut down, let it be for a coherent reason

scottaaronson.blog

351–360 of 475 posts

Re: If AI scaling is to be shut down, let it be for a coherent reason

#351

prompt: what is the third word in this sentence gpt4: third why are we afraid of this thing exactly? sure, it will improve, especially with the plugin story, but the fundamental shortcomings that underpins how it actually works won't go away anytime soon. Many people are spooked, because for the first time ever a computer can somewhat coherently understand and output natural language. As Stephen Wolfram pointed out,…

Depends on how you measure thirdiness. Is ordinal more important than actually matching the letters?

Re: If AI scaling is to be shut down, let it be for a coherent reason

#352

I read the times article, and I also watched Lex Fridman's talk with Eliezer Yudkowsky. Frankly I don't think he is qualified, and I don't understand why anyone is even giving him any credence. His argument is literally: > A mis-aligned super AGI will result in the death of humanity I ask, how . He is making reductive logical leaps that don't make sense. It's FUD.

>I ask, how. A superintelligent AGI could easily follow this three step plan: 1. Optional: Overtake and spread computation to security vulnerable computers (presumably, basically every computer) 2. Gain a physical presence by convincing humans to build critical physical components. For example by sending them emails and paying them for it. 3. Use that presence to start a grey-goo like world takeover through replicati…

> Overtake and spread computation to security vulnerable computers (presumably, basically every computer)

You could backdoor computers, sure. Spread your own computation to them? You just can't get a better-than-GPT-4 model to run at real-time speeds decentralized over wide area networks. Literally impossible. There's not the bandwidth, not the local compute hardware, and no access to specialized inference hardware.

> Gain a physical presence by convincing humans to build critical physical components. For example by sending them emails and paying them for it.

Pay for it using what money?

> Use that presence to start a grey-goo like world takeover through replicating assemblers (they don't have to be tiny)

As someone who actually works on this, you have no idea what you are talking about.

1. Grey-goo scenarios are pure science fiction that were NEVER feasible, and known to be impossible even back in the 80's when the media misunderstood Drexler's work and ran with this half-baked idea. For a full treatment, see Drexler's own retrospective in his more recent book, Radical Abundance.

2. Nanotechnology is an extremely hard problem that is not in the slightest bit bottlenecked by compute power or intelligence capability. The things that are hard in achieving atomically precise manufacturing are not things that you can simulate on a classical computer (so a years-long R&D process is required to sort out), and there is no way to train an ML model to make better predictions without that empirical data.

People like Yudkowsky talk about AIs ordering genome sequences from bio labs and making first-generation nanotechnology by mixing chemicals in a test tube. This is pure fantasy and reflects badly on them as it shows how willing they are to generalize based on fictional evidence.

Re: If AI scaling is to be shut down, let it be for a coherent reason

#353

Earlier quoted context omitted.

> Get any Truman show vibes? No. It's an interesting thought experiment but it changes nothing. Not for me, anyway. Commenting on these threads is an activity that I do for my own entertainment and "mental flexing." If it turns out that I'm not talking to real people then it doesn't make much of a difference because I don't actually perceive these messages as anything other than pixels on a screen anyway. I hope that…

> It matters a lot to me because the whole point of commenting here (or anywhere) is to talk to other humans, not just to talk to myself. That's fair, but one of my points was that even prior to ChatGPT the ability existed for you to be "sucked into" a relationship with another under false pretenses. LLMs might make it easier to put this sort of thing on "autopilot", but if what you seek is a guarantee of talking to…

> one of my points was that even prior to ChatGPT the ability existed for you to be "sucked into" a relationship with another under false pretenses

That's true, of course, but it's still interacting with a real human being. An adverse interaction, but at least a human.

> I don't see how, in a post-LLM-world, that can't be done.

I don't see how it can be done without losing much of the value of online interactions.

> also try going out and meeting people in the real world more.

I go out and meet real people plenty, thank you very much. But we're talking about online interactions here. There should be room for people online, too.

Re: If AI scaling is to be shut down, let it be for a coherent reason

#354
post #323

Earlier quoted context omitted.

>I ask, how. A superintelligent AGI could easily follow this three step plan: 1. Optional: Overtake and spread computation to security vulnerable computers (presumably, basically every computer) 2. Gain a physical presence by convincing humans to build critical physical components. For example by sending them emails and paying them for it. 3. Use that presence to start a grey-goo like world takeover through replicati…

Yeah, this is Yud’s argument, but I just don’t get it. Does the technology to end the world by sending a couple emails around already exist? If so, why hasn’t the world ended?

No, it doesn't. If making nanotech was that easy I guarantee you others (including myself) would have done it ages ago.

Re: If AI scaling is to be shut down, let it be for a coherent reason

#355

Am I the only one who's not very concerned about ChatGPT and "AI" in general? I am young, but still lived through several hype phases. I remember when 3D TVs were going to be mainstream and 2D was considered legacy. I remember when PC was to die soon and to be replaced by smartphones. I remember when VR was to become as common accessory as a game controller. It's 2023, and I still don't have automated self driving ca…

The thing about predicting doom is that you're wrong often--until one day you're not. Most hype cycles involve people at the top making wild claims that fail to deliver. I don't know anyone outside the industry who ever thought 3D TVs were worth anything, and barely anyone who thought VR was worth anything. Google pitched AI making appointments, but that never made it off a stage. Hype? Only for some definition of hy…

> There's no question that developers using a nice IDE to write high-level code on a large monitor are able to product more code more quickly than someone writing assembler on a dumb terminal, I hope.

This is true. It's also true that the code they produce is of lower quality. In practice, for the most part, this doesn't matter because the industry has decided that it's more economical to make up for poor quality code with more performant hardware.

Re: If AI scaling is to be shut down, let it be for a coherent reason

#356
post #276

Earlier quoted context omitted.

> If an AI gets advanced enough to improve itself, it seem entirely reasonable that it would go from laughable to godlike in a week. I see this stated a lot these days but it's not true. You're imagining a exponential phenomenon, improvement leads to greater improvement, leading to still greater improvement, etc. However, all exponential phenomenon require the right environment to sustain and, by their nature, consum…

Sydney Bing taught itself to play chess without ever explicitly being told to learn chess. So yes, (1) is already occurring. GPT-4 displays emergent capabilities, one of which is generalized "learning".

So why aren't we all paperclips?

Re: If AI scaling is to be shut down, let it be for a coherent reason

#357
post #10

This whole thing is like leaving 20 kids alone in a room with a marshmallow each and telling them don't eat it. ...and then expecting all of them resist. The debate around whether we should tell the kids not to eat it and for what reasons is completely academic. Practically this just isn't happening.

It more like asking the US government to get involved and if one of the 20 kids keeps on eating marshmallows, large numbers of federal agents raid the kid's offices and shut down the kid's compute resources.

>It more like asking the US government to get involved

US gov has direct jurisdiction over 4% of the worlds population and some pretty decent indirect influence over the other 96%.

It's good, but nowhere enough to prevent secret marshmallow eating on a global scale. Not even close

Re: If AI scaling is to be shut down, let it be for a coherent reason

#358
post #158

Earlier quoted context omitted.

> Extrapolating from the last two years Therein lies the error. People forget reality is finite and just because it improved now doesn't mean it will continue indefinitely. An exponential curve is just a sigmoid curve in disguise. Most AI I've seen suffer from catastrophic errors in the tail end (famous example of two near identical cat pictures classified as cat and dog respectively).

Why is that an error? This is very new tech, if anything the rate of change is accelerating and nowhere near slowing down. That it will flatten at some point goes without saying (you'd hope!), but for the time being it looks like we're on the increasing first derivative part of the sigmoid, not on the flattening part, and from two years to a decade takes you roughly from '2 kg luggable' to 'iphone' and that was a sea…

> Why is that an error?

First. It seems like these AI models depend on the underlying hardware acceleration doubling, which is not really the case anymore.

Second. All AI's I've seen suffer from the same works fine until it just flips the fuck out behavior (and starts hallucinating). You wouldn't tollerate a programmer that worked fine except he would ocassionally come to work high enough on bath salts to start claiming the sky is red and aliens have inflitrated the Wall of China. AI's that don't suffer from this aren't general purpose.

Third. I'm not convinced in we'll make AI whose job will be to make smarter AI which will make smarter AI argument. A smart enough AI could just rewire its reward mechanism to get reward without work (or collude with other AIs meant to monitor it to just do nothing).

Re: If AI scaling is to be shut down, let it be for a coherent reason

#359

It seems as though Scott just rejects the idea of the singularity entirely. If an AI gets advanced enough to improve itself, it seem entirely reasonable that it would go from laughable to godlike in a week. I don’t know if the singularity is near, inevitable at some point, or really even possible, but it does seem like something that at least could happen. And if it occurs, it will look exactly as what he describes n…

> it seem entirely reasonable that it would go from laughable to godlike in a week

Do you realize how ridiculous this is?

Re: If AI scaling is to be shut down, let it be for a coherent reason

#360

Earlier quoted context omitted.

A mis-aligned super AGI will treat the Earth and everything on it as a playground of atoms and material. Why wouldn't it? What do children do when they see a sandbox? Do they care what happens to any ants or other bugs that might live in it? There does not need to be a "how", as you put it. The logic is "Maybe we should tread carefully when creating an intelligence that is magnitudes beyond our own". The logic is "Ma…

Most bugs haven't gone extinct though. I doesn't seem obvious that any project the AGI will find worthwhile would necessitate exterminating the humanity.

> Most bugs haven't gone extinct though.

We're working on it as fast as we can:

https://en.wikipedia.org/wiki/Decline_in_insect_populations

https://www.npr.org/sections/goatsandsoda/2022/02/24/1082752...

https://www.reuters.com/graphics/GLOBAL-ENVIRONMENT/INSECT-A...

Post reply on HN