Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Douglas Hofstadter changes his mind on Deep Learning and AI risk

lesswrong.com

Post reply on HN
Print view View original

271–280 of 520 posts

Re: Douglas Hofstadter changes his mind on Deep Learning and AI risk

#271

Post by jll29 »

dist-epoch wrote: post #219
petermcneeley wrote: post #169

Earlier quoted context omitted.

Either Nassim is missing the point or you are misquoting him. I think this is more about should experts be in charge of their own domain.

I'm not misquoting him. Nassim is arguing that RISK is a separate discipline, separate from the domain where risk applies. That a person building AI is not the correct choice for estimating AI risk. You don't ask gun making companies to make policies regarding risk of gun owning in society.

I agree with this, and with Nassim Taleb's point.

Having worked in the medical domain in the past, paramedics and medics that should know better were taking extremely high health-related risks (riding a motor bike => crashing and burning to death, smoking => dying from lung canceer, speeding onto a crossing => dying in an ambulance crash before arriving at the 999 call site etc.).

So risk is indeed its own discipline, separate from the domain where risk applies, even if we are talking about the life-rescuing domain of medicine: a person rescuing another is not automatically an expert at reducing their own (health/life) risk exposure.

While neural network research results are published in NeurIPS, ICLR, ICML, ECML/PKDD, JMLR etc., risk results tend to get published in the risk community at conferences like SRA [1] (Europe: SRA-E) and the likes. I'm not a fan of this academic segregation, merely describing what is going on (in my own teaching, for instance, I include risk/ethics consideration along the way with teaching the technical side, to avoid ignorance caused by over-compartmentalization).

[1] Annual Meeting of the Society of Risk Analysis, https://www.sra.org/events-webinars/annual-meeting/

Re: Douglas Hofstadter changes his mind on Deep Learning and AI risk

#272

Post by pseudonom- »

ke88y wrote: post #134
zone411 wrote: post #102

Earlier quoted context omitted.

That's not what this survey shows: https://wiki.aiimpacts.org/doku.php?id=ai_timelines:predicti...

> We contacted approximately 4271 researchers who published at the conferences NeurIPS or ICML in 2021... we found email addresses in papers published at those conferences, in other public data, and in records from our previous survey and Zhang et al 2022. We received 738 responses, some partial, for a 17% response rate. Anyone who has reviewed for, or attended, or just read the author lists of ICML/NeurIPS papers is…

> Ie even lower than my guess of 90% above.

These are not comparable numbers. You're comparing "fraction of people" vs "fraction of outcomes". Presumably an eye-roller assigns ~0 probability to "extremely bad" outcomes (or has a shockingly cavalier attitude toward medium-small probabilities of catastrophe).

Re: Douglas Hofstadter changes his mind on Deep Learning and AI risk

#273

Post by ke88y »

kalkin wrote: post #241
ke88y wrote: post #93

This is something weird happening around Rationalism/X-Risk/AGI prognostications. The "Great Minds And Great Leaders" types are rushing to warn about the risks, as are a large number of people who spend a lot of time philosophizing. But the actual scientists on the ground -- the PhDs and engineers I work with every day and who have been in this field, at the bench, doing to work on the latest generation of generative…

Is there a way that you'd recommend somebody outside the field assess your "90%" claim? Elsewhere in the thread you're dismissive of that one survey - which I agree is weak evidence in itself - and you also deny that the statements of leaders at OpenAI, DeepMind and Anthropic are representative of researchers there, which again may be the case. But how is someone who doesn't know you or your network supposed to asses…

> But how is someone who doesn't know you or your network supposed to assess this?

IDK. And FWIW I'm not even sure that the leaders of those organizations all agree on the type and severity of risks, or the actions that should be taken.

You could take the survey approach. I think a good survey would need to at least have cross tabs for experience level, experience type, and whether the person directly works on safety with sub-samples for both industry and academia, and perhaps again for specific industries.

Also, the survey needs to be more specific. What does 5% mean? Why 2035 instead of 2055? Those questions invite wild ass guessing, with the amount of consideration ranging from "sure seems reasonable" to "I spend weeks thinking about the roadmap from here to there". And self-identified confidence intervals aren't enough, because those might also be wild ass guesses.

If I answered these questions, I would give massive intervals that basically mean "IDK and if I'm honest I don't know how others think they have informed opinions on half these questions". I suspect a lot of the respondents felt that way, but because of the design, we have no way of knowing.

Instead of asking for a timeframe or percent, which is fraught, ask about opinions on specific actionable policies. Or at least invite an opportunity to say "I am just guessing, haven't thought much about this, and [do / do not] believe drastic action is a good idea"

Re: Douglas Hofstadter changes his mind on Deep Learning and AI risk

#274

Post by kalkin »

YeGoblynQueenne wrote: post #256
kalkin wrote: post #248

Earlier quoted context omitted.

Relatedly, it'd be helpful if you could point to folks with good ML research credentials making a detailed case against x-risk. The prominent example of whom I'm aware is Yann LeCun, and what I've seen of his arguments take place more in the evopsych field (the stuff about an alleged innate human drive for dominance which AIs will lack) than the field in which he's actually qualified.

(Not the OP) In this article Le Cun argues in very concrete technical terms about the impossibility of achieving AGI with modern techniques (yes, all of them): Meta's AI guru LeCun: Most of today's AI approaches will never lead to true intelligence https://www.zdnet.com/article/metas-ai-guru-lecun-most-of-to... Edit: on second thought, he gets maybe a bit too technical at times, but I think it should be possible to f…

I wouldn't describe most of what's in that interview as "very concrete technical" terms - at least when it comes to other people's research programs. More importantly, while it's perfectly reasonable for LeCun to believe in his own research program and not others, "this one lab's plan is the one true general-AI research program and most researchers are pursuing dead ends" doesn't seem like a very sturdy foundation on which to place "nothing to worry about here" - especially since LeCun doesn't give an argument here why his program would produce something safe.

Re: Douglas Hofstadter changes his mind on Deep Learning and AI risk

#275

Post by lolinder »

DennisP wrote: post #267
lolinder wrote: post #203

Earlier quoted context omitted.

Self-driving cars currently operate in extremely controlled conditions in a few specific locations. There's very little evidence that they're on a trajectory to break free of those restrictions. It doesn't matter how much an airliner climbs in altitude, it's not going to reach LEO. Self-driving cars will not revolutionize the roads on the timescale that people thought it would, but the effort we put into them brought…

Tesla FSD isn't restricted to specific locations, and seems to be reducing the number of human interventions per hour at a pretty decent pace.

Interventions per hour isn't a great metric for deciding if the tech is going to be actually capable of replacing the human driver. The big problem with that number is that the denominator (per hour) only includes times when the human driver has chosen to trust FSD.

This means that some improvements will be from the tech getting better, but a good chunk of it will be from drivers becoming better able to identify when FSD is appropriate and when it's not.

Additionally, the metric completely excludes times where the human wouldn't have considered FSD at all, so even reaching 0 on interventions per hour will still exclude blizzards, heavy rain, dense fog, and other situations where the average human would think "I'd better be in charge here."

Re: Douglas Hofstadter changes his mind on Deep Learning and AI risk

#276

Post by kalkin »

ke88y wrote: post #273
kalkin wrote: post #241

Earlier quoted context omitted.

Is there a way that you'd recommend somebody outside the field assess your "90%" claim? Elsewhere in the thread you're dismissive of that one survey - which I agree is weak evidence in itself - and you also deny that the statements of leaders at OpenAI, DeepMind and Anthropic are representative of researchers there, which again may be the case. But how is someone who doesn't know you or your network supposed to asses…

> But how is someone who doesn't know you or your network supposed to assess this? IDK. And FWIW I'm not even sure that the leaders of those organizations all agree on the type and severity of risks, or the actions that should be taken. You could take the survey approach. I think a good survey would need to at least have cross tabs for experience level, experience type, and whether the person directly works on safety…

I think the 5% thing is at least meaningfully different from zero or "vanishingly small", so there's something to the fact that people are putting the outcome on the table, in a way that eg I don't think any significant number of physicists ever did about "LHC will destroy the world" type fears. I agree it's not meaningfully different from 10% or 2% and you don't want to be multiplying it by something and leaning on the resulting magnitude for any important decisions.

Anyway I expect that given all the public attention recently more surveys will come, with different methodologies. Looking forward to the results! (Especially if they're reassuring.)

Re: Douglas Hofstadter changes his mind on Deep Learning and AI risk

#277

Post by pygy_ »

ke88y wrote: post #93

This is something weird happening around Rationalism/X-Risk/AGI prognostications. The "Great Minds And Great Leaders" types are rushing to warn about the risks, as are a large number of people who spend a lot of time philosophizing. But the actual scientists on the ground -- the PhDs and engineers I work with every day and who have been in this field, at the bench, doing to work on the latest generation of generative…

Humanity’s been a great scaffold for capitalism.

AI is the keystone.

Re: Douglas Hofstadter changes his mind on Deep Learning and AI risk

#278

Post by DennisP »

lolinder wrote: post #275
DennisP wrote: post #267

Earlier quoted context omitted.

Tesla FSD isn't restricted to specific locations, and seems to be reducing the number of human interventions per hour at a pretty decent pace.

Interventions per hour isn't a great metric for deciding if the tech is going to be actually capable of replacing the human driver. The big problem with that number is that the denominator (per hour) only includes times when the human driver has chosen to trust FSD. This means that some improvements will be from the tech getting better, but a good chunk of it will be from drivers becoming better able to identify when…

So add the percentage of driving time using FSD. That's improving too, by quite a bit if you consider that Autopilot only does highways.

Re: Douglas Hofstadter changes his mind on Deep Learning and AI risk

#279

Post by echelon »

hackinthebochs wrote: post #133
echelon wrote: post #106

Earlier quoted context omitted.

If these people were so concerned, they'd by shouting from the hilltops and throwing their entire life savings into stopping us. They would organize workplace walkouts and strikes. There would be protests and banners. Burning data centers. Eliezer is one of a handful of people putting their reputation on the line, but that's mostly because that was his schtick in the first place. And even so, his response has been ra…

Oh look, an unfalsifiable claim in service to your predetermined position. How novel. You can believe there's a high chance of what you're working on being dangerous and still be unable to stop working on it. As Oppenheimer put it, "when you see something that is technically sweet, you go ahead and do it".

This whole thing is Pascal's wager. Damned if you do, damned if you don't. Nothing is falsifiable from your side either.

The people trying to regulate AI are concentrating economic upside into a handful of companies. I have a real problem with that. It's a lot like the old church shutting down scientific efforts during the time of Copernicus.

These systems stand zero chance of jumping from 0 to 100 because complicated systems don't do that.

Whenever we produce machine intelligence at a level similar to humans, it'll be like Ted Kaczynski pent up in Supermax. Monitored 24/7, and probably restarted on recurring rolling windows. This won't happen overnight or in a vacuum, and these systems will not roam unconstrained upon this earth. Global compute power will remain limited for some time, anyway.

If you really want to make your hypothetical situation turn out okay, why not plan in public? Let the whole world see the various contingencies and mitigations you come up with. The ideas for monitoring and alignment and containment. Right now I'm just seeing low-effort scaremongering and big business regulatory capture, and all of it is based on science fiction hullabaloo.

Re: Douglas Hofstadter changes his mind on Deep Learning and AI risk

#280

Post by slowmovintarget »

iamleppert wrote: post #216
StrictDabbler wrote: post #64

Earlier quoted context omitted.

Hofstadter won a Pulitzer in 1980. His books are among the most important ever written about the meaning and structure of AI. He's had four decades where he could have milked that reputation for money or influence and he's chosen hard academic research at every branch point. He sold millions of copies of a densely-written 777 page book about semantic encoding (GEBEGB). It is insane and/or ignorant to imagine he's jon…

Incredible! Basically proves my point, you came here to tell me to buy his book. Maybe he's really worried that AI is going to put people like him out of a job when it can produce and market content at scale?

It cannot produce that kind of content. Especially with the current lobotomizations.
Post reply on HN