Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Douglas Hofstadter changes his mind on Deep Learning and AI risk

lesswrong.com

Post reply on HN
Print view View original

131–140 of 520 posts

Re: Douglas Hofstadter changes his mind on Deep Learning and AI risk

#131

Post by dist-epoch »

ke88y wrote: post #93

This is something weird happening around Rationalism/X-Risk/AGI prognostications. The "Great Minds And Great Leaders" types are rushing to warn about the risks, as are a large number of people who spend a lot of time philosophizing. But the actual scientists on the ground -- the PhDs and engineers I work with every day and who have been in this field, at the bench, doing to work on the latest generation of generative…

Nassim Taleb makes a great point:

- The best person to judge the risk of playing roulette is not the carpenter who built it.

- The best person to judge the risks of a global pandemic is not a virologist working with viruses daily.

You can extend that:

- The best person to judge the cybersecurity risks of an application is not the programmer implementing it.

Re: Douglas Hofstadter changes his mind on Deep Learning and AI risk

#132

Post by tmsh »

ralphc wrote: post #107

When I read about the dangers of AI I'm reminded of the feeling I had after reading Jeff Hawkin's "On Intelligence". He talked about simulating the neocortex to do many of the things that deep learning and LLM's are doing now. His research may or may not be a dead end but his work, and this work, to me seems like we're building the neocortex layer without building the underlying "lizard brain" that higher animals' br…

+1 I tend to agree with that in terms of how to think about AI. It's all just neocortex. The alignment issue could be unpacked as more of a "lizard brain mammalian brain pre-frontal cortex LLM-enhanced cortex" alignment issue.

But when real self-replication starts happening -- that is maybe the really exciting/terrifying area. It's more that humans with generative AI are almost strong enough to create artificial life. And when that pops off -- when you have things trying to survive -- that's where we need to be careful. I guess I would regulate that area -- mostly around self-replication.

Re: Douglas Hofstadter changes his mind on Deep Learning and AI risk

#133

Post by hackinthebochs »

echelon wrote: post #106
zone411 wrote: post #102

Earlier quoted context omitted.

That's not what this survey shows: https://wiki.aiimpacts.org/doku.php?id=ai_timelines:predicti...

If these people were so concerned, they'd by shouting from the hilltops and throwing their entire life savings into stopping us. They would organize workplace walkouts and strikes. There would be protests and banners. Burning data centers. Eliezer is one of a handful of people putting their reputation on the line, but that's mostly because that was his schtick in the first place. And even so, his response has been ra…

Oh look, an unfalsifiable claim in service to your predetermined position. How novel.

You can believe there's a high chance of what you're working on being dangerous and still be unable to stop working on it. As Oppenheimer put it, "when you see something that is technically sweet, you go ahead and do it".

Re: Douglas Hofstadter changes his mind on Deep Learning and AI risk

#134

Post by ke88y »

zone411 wrote: post #102
ke88y wrote: post #93

This is something weird happening around Rationalism/X-Risk/AGI prognostications. The "Great Minds And Great Leaders" types are rushing to warn about the risks, as are a large number of people who spend a lot of time philosophizing. But the actual scientists on the ground -- the PhDs and engineers I work with every day and who have been in this field, at the bench, doing to work on the latest generation of generative…

That's not what this survey shows: https://wiki.aiimpacts.org/doku.php?id=ai_timelines:predicti...

> We contacted approximately 4271 researchers who published at the conferences NeurIPS or ICML in 2021... we found email addresses in papers published at those conferences, in other public data, and in records from our previous survey and Zhang et al 2022. We received 738 responses, some partial, for a 17% response rate.

Anyone who has reviewed for, or attended, or just read the author lists of ICML/NeurIPS papers is LOLing right now. Being on the author list of an ICML/NeurIPS paper does not an expert make.

Anyone who has tried to get a professor or senior researcher to answer an email -- let alone open an unsolicited email and take a long ass survey -- is laughing even harder.

I think their methodology almost certainly over-sampled people with low experience and high exuberance (ie, VERY young scientists still at the beginning of their training period and with very little technical or life experience). You would expect this population, if you have spent any time in a PhD student lounge or bar near a conference, to RADICALLY over-estimate technological advancement.

But even within that sample:

> The median respondent believes the probability that the long-run effect of advanced AI on humanity will be “extremely bad (e.g., human extinction)” is 5%.

Ie even lower than my guess of 90% above.

Re: Douglas Hofstadter changes his mind on Deep Learning and AI risk

#135

Post by yanderekko »

echelon wrote: post #106
zone411 wrote: post #102

Earlier quoted context omitted.

That's not what this survey shows: https://wiki.aiimpacts.org/doku.php?id=ai_timelines:predicti...

If these people were so concerned, they'd by shouting from the hilltops and throwing their entire life savings into stopping us. They would organize workplace walkouts and strikes. There would be protests and banners. Burning data centers. Eliezer is one of a handful of people putting their reputation on the line, but that's mostly because that was his schtick in the first place. And even so, his response has been ra…

>If these people were so concerned, they'd by shouting from the hilltops and throwing their entire life savings into stopping us. They would organize workplace walkouts and strikes. There would be protests and banners. Burning data centers.

I think we underestimate the intoxicating lure of human complacency at our own peril. If I think there's a 90% chance that AI will kill me in the next 20 years, maybe I'd be doing this. Of course, there is a knowing perception that appearing too unhinged can be detrimental to the cause, eg. actually instigating terrorist attacks against AI research labs may backfire.

But if I only think there's a 25% chance? Ehh. My life will be less-stressed if I don't think about it too much, and just go on as normal. I'm going to die eventually anyways, if it's part of a singularity event then I imagine it will be quick and not too painful.

Of course, if the 25% estimate were accurate, then it's by far the most important policy issue of the current day.

Also of course there are collective action problems. If I think there's a 90% chance AI will kill me, then do I really think I can bring that down appreciably? Probably not. I could probably still have a bigger positive impact on my life expectancy by say dieting better. And let's see how good humans are at that..

Re: Douglas Hofstadter changes his mind on Deep Learning and AI risk

#136

Post by dist-epoch »

ralphc wrote: post #107

When I read about the dangers of AI I'm reminded of the feeling I had after reading Jeff Hawkin's "On Intelligence". He talked about simulating the neocortex to do many of the things that deep learning and LLM's are doing now. His research may or may not be a dead end but his work, and this work, to me seems like we're building the neocortex layer without building the underlying "lizard brain" that higher animals' br…

> We tell them what to do. The AIs by themselves don't need to be feared, we need to fear what people with lizard brains use them for.

Even if so, there will always be that one person which wants to see the world burn.

If suddenly every person in SF had a nuclear bomb, how long do you think it would take until someone presses the button? I bet less than 5 minutes.

Re: Douglas Hofstadter changes his mind on Deep Learning and AI risk

#138

Post by echelon »

zone411 wrote: post #128
echelon wrote: post #106

Earlier quoted context omitted.

If these people were so concerned, they'd by shouting from the hilltops and throwing their entire life savings into stopping us. They would organize workplace walkouts and strikes. There would be protests and banners. Burning data centers. Eliezer is one of a handful of people putting their reputation on the line, but that's mostly because that was his schtick in the first place. And even so, his response has been ra…

If you read the survey, you'll find that many concerned researchers don't believe we're at a 90% chance of doom, but e.g. 10%. So, this type of response wouldn't be rational if they're thinking that things will go fine most of the time. If these researchers are thinking logically, they would also realize that this kind of reaction has little chance of success, especially if research continues in places like China. It…

> If you read the survey

Not a nice way to engage in debate. I've spent more time listening to and refuting these arguments than most. Including debating Eliezer.

> many concerned researchers don't believe we're at a 90% chance of doom, but e.g. 10%.

A 10% chance of an asteroid hitting the earth would result in every country in the world diverting all of their budgets into building a means to deflect it.

> So, this type of response wouldn't be rational.

This is the rational response to a 10% chance?

These are funny numbers and nobody really has their skin in the game here.

If I believed (truly believed) their arguments, I would throw myself at stopping this. Nobody is doing anything except for making armchair prognostications and/or speaking to congress as an AI CEO about how only big companies such as their own should be doing AI.

> especially if research continues in places like China.

I like how both sides of this argument are using the specter of China as a means to further their argument.

Re: Douglas Hofstadter changes his mind on Deep Learning and AI risk

#139

Post by hackinthebochs »

jrflowers wrote: post #114
hackinthebochs wrote: post #77

Earlier quoted context omitted.

Seeing how confidently and obtusely people dismiss the risks of AI makes me so much more fearful of what's to come. Human hubris knows no limits.

It’s interesting how confidently and obtusely people will proclaim categorical knowledge of the future. It is a little disconcerting that there is a fight between two somewhat cultish sects when it comes to language models. Both sides call them “artificial intelligence”, one side says they’ll save the world, the other side says they’ll end it. There is very little room to even question “Is this actually AI that we’re…

Trying to impute the motives of ones interlocutor is dumb and boring. How about we discuss the first-order issue instead. Here's my argument for why x-risk is a real possibility:

The issue is that small misalignments in objectives can have outsized real-world effects. Optimizers are constrained by rules and computational resources. General intelligence allows an optimizer to find efficient solutions to computational problems, thus maximizing the utility of available computational resources. The rules constrain its behavior such that on net it ideally provides sufficient value to us above what it destroys. But misalignment in objectives provides an avenue by which the AGI can on net destroy value despite our best efforts. Can you be sure you can provide loophole-free objectives that ensures only value-producing behavior from the human perspective? Can you prove that the ratio of value created to value lost due to misalignment is always above some suitable threshold? Can you prove that the range of value destruction is bounded so that if it does go off the rails, its damage is limited? Until we do, x-risk should be the default assumption.

What say you?

Post reply on HN