Douglas Hofstadter changes his mind on Deep Learning and AI risk • Hacker News Board

Re: Douglas Hofstadter changes his mind on Deep Learning and AI risk

#121

Post by jameshart » Mon, Jul 03, 2023, 6:09 PM UTC

jackconsidine wrote: post #2

In GEB Hofstadter dismisses the idea that AI could understand / compose / feel music like a human. I thought about this a lot when I started using GPT, especially early on when it demonstrated an ability to explain why things were funny or sad, intrinsically human qualities hitherto insulated from machine

I don’t know I’d agree that that was the message of GEB at all. In fact more than anything GEB and I am a Strange Loop convincingly argue that consciousness, understanding, and feeling arise from systems that are no more complex than an information system that feeds on its own output. Though he is troubled by what kind of feedback it is that is required to make that loop into a mind.

Hofstadter is why I am not sure why AI researchers feel so confident in saying ‘LLMs can’t be thinking, they’re just repeatedly generating the next token’ - I don’t think there’s any evidence that you need anything more complicated than that to make a mind, so how can you be certain you haven’t?

GEB may have been dismissive of the idea that the approaches that were being taken in AI research at the time were likely to result in intelligence - but I don’t think GEB is pessimistic about the possibility of artificial consciousness at all.

Re: Douglas Hofstadter changes his mind on Deep Learning and AI risk

#122

Post by ke88y » Mon, Jul 03, 2023, 6:09 PM UTC

startupsfail wrote: post #105

ke88y wrote: post #93

This is something weird happening around Rationalism/X-Risk/AGI prognostications. The "Great Minds And Great Leaders" types are rushing to warn about the risks, as are a large number of people who spend a lot of time philosophizing. But the actual scientists on the ground -- the PhDs and engineers I work with every day and who have been in this field, at the bench, doing to work on the latest generation of generative…

Don’t forget that most of these engineers down in the trenches can’t see further than their battlefield. You actually need someone with vision and track record of doing right predictions and placing right technology bets. Ask engineers that had placed their bet on deep learning and generative models back when discriminative models and support vector machines were a rage of dat, a few years before Alexnet (I’m one of…

> Don’t forget that most of these engineers down in the trenches can’t see further than their battlefield.

I don't think this is the case at all. I'm not primarily talking about a junior or even senior engineer with a decade of experience working on product features. On the contrary, many of these people have PhDs, have been leading research agendas in this field for decades, have been in senior leadership roles for a long time, have launched successful products, etc. etc.

> Ask engineers that had placed their bet on deep learning and generative models back when discriminative models and support vector machines were a rage of dat, a few years before Alexnet (I’m one of such engineers). I’d bet the answer will be different.

And at that time half of the "Great Minds And Great Leaders" prognosticating on X-Risk were doing social web or whatever else was peak hype cycle back then.

Re: Douglas Hofstadter changes his mind on Deep Learning and AI risk

#123

Post by jameshart » Mon, Jul 03, 2023, 6:10 PM UTC

gamegoblin wrote: post #109

jameshart wrote: post #75

Interesting to hear him say this: > And I would never have thought that deep thinking could come out of a network that only goes in one direction, out of firing neurons in only one direction. And that doesn't make sense to me, but that just shows that I'm naive. I think people maybe miss that LLM output does involve a ‘loop’ back - maybe even a ‘strange’ loop back, and I’m surprised to see Hofstadter himself fail to…

Just being a part of any auto-regressive system does not contradict his statement. Go look at the GPT training code, here is the exact line: https://github.com/karpathy/nanoGPT/blob/master/train.py#L12... The model is only trained to predict the next token . The training regime is purely next-token prediction. There is no loopiness whatsoever here, strange or ordinary. Just because you take that feedforward neural ne…

It only yields something that looks like intelligence when you update the context and iterate though.

Re: Douglas Hofstadter changes his mind on Deep Learning and AI risk

#124

Post by gamegoblin » Mon, Jul 03, 2023, 6:11 PM UTC

ke88y wrote: post #110

gamegoblin wrote: post #57

Earlier quoted context omitted.

All the other responses to you at the time of writing this comment are confidently wrong. Definition of Feedforward (from wiki): ``` A feedforward neural network (FNN) is an artificial neural network wherein connections between the nodes do not form a cycle.[1] As such, it is different from its descendant: recurrent neural networks. ``` Hofstadter expected any intelligent neural network would need to be recurrent, ie…

Right. I'm not at all sure what the siblings are talking about. I suspect at least one is confusing linear with feed-forward? But I'm also surprised that Hofstadter keys in on this so heavily. The fact that he wrote an entire pop-sci book on recursion would, in my mind, make him (1) less surprised that AR and R aren't so dissimilar and (2) more sensitive to the sorts of issues that make R more difficult to get workin…

I think it has to do with the training regime and fixed-computation time nature of feedforward neural networks.

Recurrent neural networks have the recursion as part of the training regime. GPT only has auto-regressive "recursion" as part of the inference runtime regime.

I think Hofstadter is surprised that you can appear so intelligent without any recursion in the learning/training regime, with the added implication that you can appear so intelligent with a fixed amount of computation per word.

Re: Douglas Hofstadter changes his mind on Deep Learning and AI risk

#125

Post by billti » Mon, Jul 03, 2023, 6:13 PM UTC

jurgenaut23 wrote: post #9

I read Hofstadter's GEB and Tegmark's Our Mathematical Universe, and of course I developed a rather fond admiration of these brilliant minds. For some reason, both of them have developed a profound aversion and fear of what they consider an existential threat. I have a solid theoretical understanding of these systems, and I spent 15 years studying, building, and deploying them at scale and for diverse use cases. The…

I don't think many people are worried that what we have now is a major risk. The major concern is the implications of the trajectory we are now on.

If you look at what ChatGPT and Midjourney and the like can do now compared to just a couple of years ago, it's pretty incredible. If you extrapolate the next few similar jumps in capability, and assume that won't be 20 years away, then what AI is going to be capable of before even my kids leave college is going to be mind-boggling, and in some possible futures not in a good way.

I remember seeing this talk from Sam Harris nearly 6 years ago and it logically making a lot of sense back then (https://youtu.be/8nt3edWLgIg). The past couple of years have made this all the more prescient. (Worth a watch if you have 15 mins).

Re: Douglas Hofstadter changes his mind on Deep Learning and AI risk

#126

Post by ke88y » Mon, Jul 03, 2023, 6:13 PM UTC

hnaccy wrote: post #119

ke88y wrote: post #93

This is something weird happening around Rationalism/X-Risk/AGI prognostications. The "Great Minds And Great Leaders" types are rushing to warn about the risks, as are a large number of people who spend a lot of time philosophizing. But the actual scientists on the ground -- the PhDs and engineers I work with every day and who have been in this field, at the bench, doing to work on the latest generation of generative…

> Altman's criticisms of criticisms about regulatory capture What is his criticism? If you agree with silent majority who seem to think it's not dangerous why agree with Altman who rants regulation.

> What is his criticism?

I heard in some interview, I think with Bloomberg, where he said that claims about regulatory capture were "so disingenuous I'm not sure what to say", or something like that.

I think he's probably not lying when he says that his goal isn't regulatory capture (although I do think other people perceiving that to be his intent aren't exactly insane either...)

> who seem to think it's not dangerous

On the contrary. They think it's dangerous but in a more mundane way, and that the X-Risk stuff is idiotic. I tend to agree.

> why agree with Altman who rants regulation

IDK. What even are his proposed regulations? They're so high-level atm that they could literally mean anything.

In terms of the senate hearing he was part of, and what the government should be doing in the near term, I think the IBM woman was the only adult in the room regarding what should actually be done over the next 3-5 years.

But her recommendations were boring and uninteresting recommendations to do basically the exactly sort of mundane shit the wheels of government tend to do when a new technology arrives on the scene, instead of breathless warnings about killer AI, so everyone brushed her off. But I think she's more or less right -- what should we do? The same old boring shit we always do with any new technology.

Re: Douglas Hofstadter changes his mind on Deep Learning and AI risk

#127

Post by petermcneeley » Mon, Jul 03, 2023, 6:13 PM UTC

ke88y wrote: post #93

This is something weird happening around Rationalism/X-Risk/AGI prognostications. The "Great Minds And Great Leaders" types are rushing to warn about the risks, as are a large number of people who spend a lot of time philosophizing. But the actual scientists on the ground -- the PhDs and engineers I work with every day and who have been in this field, at the bench, doing to work on the latest generation of generative…

They are building consensus and finding alignment. The problem is power bends truth. This is all about access to a new powerful tool. They want to concentrate that access in the hands of those that already have control. The end goal here is the destruction of the general purpose computer.

Re: Douglas Hofstadter changes his mind on Deep Learning and AI risk

#128

Post by zone411 » Mon, Jul 03, 2023, 6:14 PM UTC

echelon wrote: post #106

zone411 wrote: post #102

Earlier quoted context omitted.

That's not what this survey shows: https://wiki.aiimpacts.org/doku.php?id=ai_timelines:predicti...

If these people were so concerned, they'd by shouting from the hilltops and throwing their entire life savings into stopping us. They would organize workplace walkouts and strikes. There would be protests and banners. Burning data centers. Eliezer is one of a handful of people putting their reputation on the line, but that's mostly because that was his schtick in the first place. And even so, his response has been ra…

If you read the survey, you'll find that many concerned researchers don't believe we're at a 90% chance of doom, but e.g. 10%. So, this type of response wouldn't be rational if they're thinking that things will go fine most of the time. If these researchers are thinking logically, they would also realize that this kind of reaction has little chance of success, especially if research continues in places like China. It's more likely that such an approach would backfire in the court of public opinion.

Re: Douglas Hofstadter changes his mind on Deep Learning and AI risk

#129

Post by gamegoblin » Mon, Jul 03, 2023, 6:14 PM UTC

jameshart wrote: post #123

gamegoblin wrote: post #109

Earlier quoted context omitted.

Just being a part of any auto-regressive system does not contradict his statement. Go look at the GPT training code, here is the exact line: https://github.com/karpathy/nanoGPT/blob/master/train.py#L12... The model is only trained to predict the next token . The training regime is purely next-token prediction. There is no loopiness whatsoever here, strange or ordinary. Just because you take that feedforward neural ne…

It only yields something that looks like intelligence when you update the context and iterate though.

GPT can give a single Yes/No answer that indicates a fair amount of intelligence for the right question. No iteration there. Just a single pass through the network. Hofstadter is surprised by this.

Re: Douglas Hofstadter changes his mind on Deep Learning and AI risk

#130

Post by JoshTko » Mon, Jul 03, 2023, 6:15 PM UTC

echelon wrote: post #106

zone411 wrote: post #102

Earlier quoted context omitted.

That's not what this survey shows: https://wiki.aiimpacts.org/doku.php?id=ai_timelines:predicti...

If these people were so concerned, they'd by shouting from the hilltops and throwing their entire life savings into stopping us. They would organize workplace walkouts and strikes. There would be protests and banners. Burning data centers. Eliezer is one of a handful of people putting their reputation on the line, but that's mostly because that was his schtick in the first place. And even so, his response has been ra…

Is this a fair test? If you are a person with average person resources and don't expect you can impact what gets built why would you jeopardize your livelihood to make no impact?