Douglas Hofstadter changes his mind on Deep Learning and AI risk

The interesting thing though is that attention effectively allows a model to meta-learn based on the current context, so in many ways, it may be thought of as analogous to a brain without long term memory.

It’s interesting how confidently and obtusely people will proclaim categorical knowledge of the future.

It is a little disconcerting that there is a fight between two somewhat cultish sects when it comes to language models. Both sides call them “artificial intelligence”, one side says they’ll save the world, the other side says they’ll end it.

There is very little room to even question “Is this actually AI that we’re looking at?” when loudest voices on the subject are VC tech bros and a Harry Potter fan fiction author that’s convinced people that he is prescient.

While this might be somewhat applicable for Hofstadter or Tegmark, I think it doesn't applies quite the same way to Hinton. Remember, until this Spring, he's been at Google Brain and has been supervising grad students until recently (famously including Ilya Sutskever (2012)), and has invested in Cohere and other AI startups. I also has a suspicion Hinton might also have a "solid theoretical understanding" of these systems, and don't think he's being misled at all.

Also, I don't think that any of these people think that GPT-4 is itself an existential threat, but rather are worried about the exponential curve of development (I listened to Tegmark's Lex Podcast interview and that seemed to be his main concern). I think it's prudent to be worrying now, especially when capabilities growth is far outstripping safety. This is a huge concern to society whether you are considering, alignment, control, or even bad actor prevention/societal upheaval.

I've also been spending most of my waking hours these past months poking at LLM models and code, and trying to keep up on the latest ML research (it's own full time job), and while I do think there's a pretty good chances AI kills us all, I think it's much more likely it's because we make some incredibly capable AIs and people will tell them to do so, rather than it being contingent on an independent super-intelligence arising and deciding to on its own (although I do think that's a non-zero risk).

As you say, I guess we'll just have to see where we top out on this particular sigmoid, but I'm all for more people thinking through the implications, because I think so far I don't think we (as a society) have thought this through very well yet and all the money and momentum is going to keep pushing along that path.

Part of the problem is that AI doesn’t need to be an AGI to cause large society level disruption.

Eg, starting a mass movement online requires a few percent of online participants to take part in the movement. That could be faked today using a lot of GPT4 agents whipping up a storm on Twitter. And this sort of stuff shapes policies and elections. With the opensource LLM community picking up steam, it’s increasingly possible for one person to mass produce this sort of stuff, let alone nation state adversaries.

There’s a bunch of things like this that we need to watch out for.

For our industry, within this decade we’ll almost certainly have LLMs able to handle the context size of a medium software project. I think it won’t be long at all before the majority of professional software engineering is done by AIs.

There’s so much happening in AI right now. H100s are going to significantly speed up learning. Quantisation has improved massively. We have lots of papers around demoing new techniques to grow transformer context size. Stable diffusion XL comes out this month. AMD and Intel are starting to seriously invest in becoming competitors to nvidia in machine learning. (It’ll probably take a few years for PyTorch to run well on other platforms, but competition will dramatically lower prices for home AI workstations.)

Academia is flooded with papers full of new methods that work today - but which just haven’t found their way into chatgpt and friends yet. As these techniques filter down, our systems will keep getting smarter.

What a time to be alive.

When I read it when it first came out, that was my reaction. I didn't understand why it was being admired.