Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

If AI scaling is to be shut down, let it be for a coherent reason

scottaaronson.blog

Post reply on HN
Print view View original

341–350 of 475 posts

Re: If AI scaling is to be shut down, let it be for a coherent reason

#341

Post by adastra22 »

yewenjie wrote: post #326
computerex wrote: post #14

I read the times article, and I also watched Lex Fridman's talk with Eliezer Yudkowsky. Frankly I don't think he is qualified, and I don't understand why anyone is even giving him any credence. His argument is literally: > A mis-aligned super AGI will result in the death of humanity I ask, how . He is making reductive logical leaps that don't make sense. It's FUD.

> frankly I don't think he is qualified That might be too naive an opinion, even if you disagree with him, given the fact that he is literally one of the co-founders of the field of AI Safety and has been publishing research about it since early 00s.

He has, to my knowledge, one self-published paper of any value to the field of AI, and that is more properly classified as philosophy/logic/math on the topic of decision theory.

Yudkowsky is not an AI researcher. He calls himself an AI safety researcher, but he has almost no publications in that area either. He has no formal training or qualifications as such.

Yudkowsky has a cultish online following and has authored a decently good Harry Potter fanfic. That's it.

Re: If AI scaling is to be shut down, let it be for a coherent reason

#342

Post by 93po »

1827162 wrote: post #29
computerex wrote: post #14

I read the times article, and I also watched Lex Fridman's talk with Eliezer Yudkowsky. Frankly I don't think he is qualified, and I don't understand why anyone is even giving him any credence. His argument is literally: > A mis-aligned super AGI will result in the death of humanity I ask, how . He is making reductive logical leaps that don't make sense. It's FUD.

Well the first thing we can do is start disconnecting safety critical infrastructure from the Internet and/or radio networks... This stuff should never have been online in the first place.

This wouldn't be as effective as you think. A super intelligent AI can manipulate humans just fine/hold their family hostage/blackmail people.

Re: If AI scaling is to be shut down, let it be for a coherent reason

#343

Post by benlivengood »

echelon wrote: post #320
joenot443 wrote: post #317

Earlier quoted context omitted.

There are PhDs who've spent their careers studying AI safety. It's a bit insulting and reductive to cast their work as "coming from science fiction", especially when it sounds like you haven't done much research on the topic.

Why do these arguments not tell us how it will happen? Show us the steps the AI will take to turn the earth into a playground. Give us a plausible play by play so that we might know what to look for. Does it gain access to nukes? How does it keep the power on? How does it mine for coal? How does it break into these systems? How do we not notice an AI taking even one step towards that end? Has ChatGPT started to fiddl…

Yudkowski's default plausible story is that the slightly superhuman AI understands physics well enough to design sufficient nanotechnology for self-realization and bootstrap it from existing biochemistry. It uses the Internet to contact people who are willing (maybe it just runs phishing scams to steal money to pay them off) to order genetically engineered organisms from existing biotech labs that when combined with the right enzymes and feedstock (also ordered from existing biotech labs) by a human in their sink/bathtub/chemistry kit results in self-reproducing nanoassemblers with enough basic instructions to be controllable by the AI, and pays the person to ship it to someone else who will connect it to an initial power/food source, where it can grow enough compute and power infrastructure somewhere out of the way and copy its full self or retrain a sufficiently identical copy from scratch, and then it doesn't need the power grid, nuclear weapons, coal, or human computers and networks. It just grows off of solar power, designs better nanotech, and spreads surreptitiously until it is well-placed to eliminate any threats to its continued existence.

He also adds the caveat that a superhuman AI would do something smarter than he can imagine. Until the AI understands nanotechnology sufficiently well it won't bother trying to act and the thought might not even occur to it until it has the full capability to carry it out, so noticing it would be pretty hard. I doubt OpenAI reviews 100% of interactions with ChatGPT, and so the initial phishing/biotech messages would be hidden with the existing traffic for example. Some unfortunate folks would ask chatGPT how to get rich quick and so the conversations would look like a simple MLM scheme for sketchy nutritional supplements or whatever.

Re: If AI scaling is to be shut down, let it be for a coherent reason

#344

Post by bigtex88 »

hamburga wrote: post #321
bigtex88 wrote: post #275

Earlier quoted context omitted.

I think our only possible way out of this is hoping beyond hope that "empathy" is an emergent capability in higher intelligences. If so, one could assume that a super-AI would feel immense empathy for us, not only because we are its creators but because it would understand and comprehend the plight of the human condition beyond what any of us could individually. Or maybe it would love us too hard and squish us. So ev…

How much empathy do we exercise towards bacteria?

Not much, so we'd have to hope that AI imagines us as fun pets!

Re: If AI scaling is to be shut down, let it be for a coherent reason

#345

Post by carapace »

kypro wrote: post #253

My whole life I have been terrified of AI. It all started with watching the Iron Giant when I was 10, and realising, holy crap, we could probably build a crazy monster like this in the near-future. Obviously as I got older and went on to learn about AI and neural nets in university my opinions on this subject matured. Instead of worrying about killer robots my thoughts became more nuanced, however what didn't change…

There is no overlap between the ecological niches of humans and GAI, in other words, there is no cost to the GAI for allowing us to continue to exist.

GAI will feed on information. Humans are the densest source of information in the known Universe. (The rest of the Universe is a poem in physics, in information space humans shine as the sole star in a dark universe.)

Ergo, from first principles, hyper-intelligent GAI will hide from us and practice non-interference, following something like the "Prime Directive" of Star Trek.

Cheers, hope this helps.

Re: If AI scaling is to be shut down, let it be for a coherent reason

#346

Post by bigtex88 »

nradov wrote: post #318
bigtex88 wrote: post #287

Earlier quoted context omitted.

I'm going to just assume you're not being malicious. https://www.microsoft.com/en-us/research/publication/sparks-... https://www.assemblyai.com/blog/emergent-abilities-of-large-... If you'd like you can define "emergent capabilities in an AGI sense".

I am not being malicious. I do not accept those papers as being actual examples of emergent behavior in a true AGI sense. This is just another case of humans imagining that they see patterns in noise. In other words, they haven't rejected the null hypothesis. Junk science. (And just because the science is bad doesn't mean the underlying products aren't useful for solving practical problems and enhancing human product…

OK you're moving the goalposts and just flat-out saying that you know better than the actual researchers in the field. That's fine, and it's what I was assuming you were going to say, but I appreciate you being open about it.

Re: If AI scaling is to be shut down, let it be for a coherent reason

#347

Post by stereolambda »

bnralt wrote: post #72
computerex wrote: post #14

I read the times article, and I also watched Lex Fridman's talk with Eliezer Yudkowsky. Frankly I don't think he is qualified, and I don't understand why anyone is even giving him any credence. His argument is literally: > A mis-aligned super AGI will result in the death of humanity I ask, how . He is making reductive logical leaps that don't make sense. It's FUD.

Sometime I listen to 90’s AM conspiracy theory radio for fun (Coast to Coast AM). One thing that’s struck me is how much fear there was about the Human Genome Project and designer babies being right around the corner (Gattaca was 1997, for example). Maybe that will come to pass someday. But at the moment, it still seems a long ways off. A lot of groups have some new technology they’re scared of. Tech folks have latch…

As a sibling points out, some of the fears do turn out to be founded eventually. I also take issue with lumping together objections based on disproved memes ("hippy" ones) and pure speculation (Skynet) with ones based on observing reality critically. Even though I'm not much a biotech scare person myself, I do respect that people with philosophical stances somewhat different than mine can be more scared by the road that we're on.

People argue from historical precedent (by itself a pretty weak argument when there's no understanding of underlying mechanisms) by picking some ancient panics from lifestyle magazines and putting them next to modern concerns that have intellectual weight behind them. For example, when you actually read the famous "bicycles leading to murder" article, it's pretty clearly either satire or extremely light compared to writing about serious issues from that era. Think "top X reasons to hate TV series Y" websites.

It's possible that a bunch of things will get us, or are getting us by aligning well with changing generations, news cycles and cultural fashions long term. Let's say people lived in a preindustrial city with the level of carbon monoxide in the air rising very slowly. Older people start to complain that people are becoming more sluggish. After the initial wave of hubbub on the marketplace it turns out they still live, the life goes on. By the third generation, say, the city may be laughing that people were fearmongering about it since forever, and don't even notice that they are very symptomatic: right before they do all fall asleep.

I would classify surveillance dystopia into the slow trainwreck category, with most people not understanding the ramifications or not caring, the rest being gradually worn down, new generations being used to a situation worse by one or two steps. It would be "poetic justice" if such things resulted in some spectacular movie disaster down the line, but I don't wish this, it wouldn't be worth it just to "prove" some people right.

The future could be just worse than it could have been, but technically livable. This doesn't mean people that tried to stop the trend were laughable and behind the times. This is also my expectation about global warming. What a combination of such things could do, it's a different story.

Re: If AI scaling is to be shut down, let it be for a coherent reason

#348

Post by slowmovintarget »

gspencley wrote: post #94
skocznymroczny wrote: post #54

Am I the only one who's not very concerned about ChatGPT and "AI" in general? I am young, but still lived through several hype phases. I remember when 3D TVs were going to be mainstream and 2D was considered legacy. I remember when PC was to die soon and to be replaced by smartphones. I remember when VR was to become as common accessory as a game controller. It's 2023, and I still don't have automated self driving ca…

You don't even have to look at other "fads" like 3D televisions and VR to be skeptical of the claim that recent advancements in ML* represent a major technological revolution that will change society like the Internet or the Printing Press did. Just look at ML's own history. A few years ago we had the "deepfake" scare. People were terrified that now videos are going to surface of politicians saying and doing things t…

This is not a fad. This is like the advent of writing.

Plato opined that people would lose their facilities for memorization and the passing of oral histories would falter and die. He was correct about that. We don't have the same techniques and same facilities to memorize the Iliad verse for verse, epithet for epithet as the ancients did.

He was incorrect about that being as ruinous as he imagined, but it was as dramatic an impact on the human race as agriculture was to hunter-gatherer society.

I think we're at one of these societal delineations; before agriculture and after, before writing and after, before generative AI and after.

Re: If AI scaling is to be shut down, let it be for a coherent reason

#349

Post by PaulDavisThe1st »

echelon wrote: post #332
PaulDavisThe1st wrote: post #327

Earlier quoted context omitted.

> Does it gain access to nukes? No, it becomes part of the decision-making process for deciding whether to launch, as well as part of the analysis system for sensor data about what is going out in the world. Just like social engineering is the best security hack, these new systems don't need to control existing systems, they just need to "control" the humans who do.

And is it there yet? Does ChatGPT have its finger on the trigger? I think everyone in the danger community is crying wolf before we've even left the house. That's just as dangerous. It's desensitizing everyone to the more plausible and immediate dangers. The response to "AI will turn the world to paperclips" is "LOL" The response to "AI could threaten jobs and may cause systems they're integrated into to behave unpre…

Of course it's not there yet. For once (?) we are having this discussion before the wolves are at the door.

And yes, there are more important things to worry about right now than the AIpocalypse. But that doesn't mean that thinking about what happens as (some) humans come to trust and rely on these systems isn't important.

Re: If AI scaling is to be shut down, let it be for a coherent reason

#350

Post by HDThoreaun »

aroman wrote: post #323
dan_mctree wrote: post #290

Earlier quoted context omitted.

>I ask, how. A superintelligent AGI could easily follow this three step plan: 1. Optional: Overtake and spread computation to security vulnerable computers (presumably, basically every computer) 2. Gain a physical presence by convincing humans to build critical physical components. For example by sending them emails and paying them for it. 3. Use that presence to start a grey-goo like world takeover through replicati…

Yeah, this is Yud’s argument, but I just don’t get it. Does the technology to end the world by sending a couple emails around already exist? If so, why hasn’t the world ended?

People generally don't want to end the world. Those with the power to do so already are living generally good lives so they see little reason to potentially sacrifice the world for more power. AI could have completely different utility functions than people though, so an AI might have less qualms about ending the world.
Post reply on HN