Douglas Hofstadter changes his mind on Deep Learning and AI risk

That's something.

> Rationalism is rational because it is rational...

In his ideology, "rational" means "the way of thinking that best lets you achieve your goals". This not self-referential. A more appropriate criticism might be "meaningless by itself". I guess the self-referential aspect is that you're supposed to think about whether or not you're thinking well. At a basic level, that sounds useful despite being self-referential, in the same way that bootstrapping is useful. The question is if course what Yudkowski makes of this basic premise, which is hard to evaluate.

The controversy about "longtermism" has two parts. The first is a disagreement about how much to discount the future. Some people think "making absolutely sure humanity survives the next 1000 years" is very important, some people think it's not that important. There's really no way to settle this question, it's a matter of preference.

The second part is about the actual estimate of how big some dangers are. The boring part of this is that people disagree about facts and models, where more discussion is the way to go if you care about the results (which you might not). However, there is a more interesting difference between people who are/aren't sympathetic to longtermism, which lies in how they think about uncertainty.

For example, suppose you make your best possible effort (maybe someone paid you to make this worthwhile for you) to predict how likely some danger is. This prediction is now your honest opinion about this likelihood because if you'd thought it was over/under-estimated, you'd adjusted your model. Suppose also that your model seems very likely to be bad. You just don't know in which direction. In this situation, people sympathetic towards longtermism tend to say "that's my best prediction, it says there is a significant risk, we have to care about it. Let's take some precautions already and keep working on the model.". People who don't like it, in the same situation, tend to say "this model is probably wrong and hence tells us nothing useful. We shouldn't take precautions and stop modeling because it doesn't seem feasible to build a good model.".

I think both sides have a point. One side would think and act as best they can, and take precautions against a big risk that's hard to evaluate. The other would prioritize actions that are likely useful and avoid spending resources on modeling if that's unlikely to lead to good predictions. I find it a very interesting question which of these ways of dealing with uncertainty is more appropriate in everyday life, or in some given circumstances .

As you rightfully point out, the "rationalist/longtermmist" side of the discussion has an inherent tendency to detach from reality and lose itself discussing the details of lots of very unrealistic scenarios, which they must work hard to counteract. The ideas naturally attract people who enjoy armchair philosophizing and aren't likely to act based on concrete consequences of their abstract framework.

First of all, hats off to him for his extraordinary display of humility in this interview. People rarely change their minds publicly, let alone hint that they no longer believe in their own past work.

However, I'm genuinely surprised that he, of all people, does sees intelligence in GPT-4 output.

I think humans are just very eager to ascribe intelligence, or personality, to a bunch of text. A text may say "I feel " and that text can easily manage to permeate through our subconsciousness. And we end up believing that that "I" is, in fact, an "I"!

We have to actively guard against this ascription. It takes a constant self-micromanaging, which isn't a natural thing to do.

Ideally, we would have some objetive measurements (benchmarks) of intelligence. Our own impressions can be too easily fooled.

I know defining (let alone measuring) intelligence is no easy task, but in absence of a convincing benchmark, I will not give credit to new claims around AI. Else it's all hype and speculation.

Well said. Fewer Terminator and War Games fantasies, more boring risk analyses. Amen.