Douglas Hofstadter changes his mind on Deep Learning and AI risk

I don’t see AI as adding to the danger.

Self-driving cars will not revolutionize the roads on the timescale that people thought it would, but the effort we put into them brought us adaptive cruise control and lane assist, which are great improvements. AI will do similar: it will fall short of our wildest dreams, but still provide useful tools in the end.

Since I AM an expert, I care a lot less about what surveys say. I have a lot more experience working on AI Safety than 99% of the ICM/NeurIPS 2021 authors, and probably close to 100% of the respondents. In fact, I think that reviewing community (ICML/NeurIPS c. 2020) is particularly ineffective and inexperienced at selecting and evaluating good safety research methodology/results. It's just not where real safety research has historically happened, so despite having lots of excellent folks in the organization and reviewer pool, I don't think it was really the right set of people to ask about AI risk.

They are excellent conferences, btw. But it's a bit like asking about cybersecurity at a theoretical CS conference -- they are experts in some sense, I suppose, and may even know more about eg cryptography in very specific ways. But it's probably not the set of people who you should be asking. There's nothing wrong with that; not every conference can or should be about everything under the sun.

So when I say evidence, I tend to mean "evidence of X-risk", not "evidence of what my peers think". I can just chat with the populations other people are conjecturing about.

Also: even in this survey, which I don't weigh very seriously, most of the respondants agree with me. "The median respondent believes the probability that the long-run effect of advanced AI on humanity will be “extremely bad (e.g., human extinction)” is 5%", but I bet if you probed that number it's not based on anything scientific. It's a throw-away guess on a web survey. What does 5% even mean? I bet if you asked most respondents would shrug, and if pressed would express an attitude closer to mine than to what you see in public letters.

Taking that number and plugging it into a "risk * probability" framework in the way that x-risk people do is almost certainly wildly misconstruing what the respondents actually think.

> "All the oil and gas engineers I work with say climate change isn't a thing." Hmm, wow, that's really persuasive!

I totally understand this sentiment, and in your shoes my personality/temperament is such that I'd almost certainly think the same thing!!!

So I feel bad about my dismissal here, but... it's just not true. The critique of x-risk isn't self-interested.

In fact, for me, it's the opposite. It'd be easier to argue for resources and clout if I told everyone the sky is falling.

It's just because we think it's cringey hype from mostly hucksters with huge egos. That's all.

But, again, I understand that me saying that isn't proof of anything. Sorry I can't be more persuasive or provide evidence of inner intent here.

> What would you consider evidence of a significant AI risk?

This is a really good question. I would consider a few things:

1. Evidence that there is wanton disregard for basic safety best-practices in nuclear arms management or systems that could escalate. I am not an expert in geopolitics, but have consulting some on safety, and I have seen exactly the opposite attitude at least in the USA. I also don't think that this risk has anything to do with recent developments in AI; ie, the risk hasn't changed much since the early-mid 2010s. At least due to first order effects of new technology. Perhaps due to diplomatic reasons/general global tension, but that's not an area of expertise for me.

2. Specific evidence that an AI System can be used to aid in the development of WMDs of any variety, particularly by non-state actors and particularly if the system is available outside of classified settings (ie, I have less concern about simulations or models that are highly classified, not public, and difficult to interpret or operationalize without nation-state/huge corp resources -- those are no different than eg large-scale simulations used for weapons design at national labs since the 70s).

3. Specific evidence that an AI System can be used to aid in the development of WMDs of any variety, by any type of actor, in a way that isn't controllable by a human operator (not just uncontrolled, but actually not controllable).

4. Specific evidence that an AI System can be used to persuade a mass audience away from existing strong priors on a topic of geopolitical significance, and that it performs substantially better than existing human+machine systems (which already include substantial amounts of ML anyways).

I am in some sense a doomer, particularly on point 4, but I don't believe that recent innovations in LLMs or diffusion have particularly increased the risk relative to eg 2016.

Isn't that how things go in so many technology fields? "Move fast and break things" pressure to deliver and money, reputation and fame involved in doing so are equally Ego-related and leading to biases that make on "laugh or roll their eyes".

That's an orthogonal issue to actual x-risk, and I covered a bit more here: https://news.ycombinator.com/item?id=36577523

> I'd personally suggest keeping an eye on white papers from RAND and the like in trying to get some sense of the actual implications in the real world

I think this is excellent advice and we're on the same page.