And yet It Understands

Comparing this to atheism is very illuminating, actually.

In some ways this topic is very different to atheism/theism. In the ‘belief in god’ debate both sides are trying to take away the other side’s belief; it’s important to atheists to stop people believing in God, while to believers it’s important to stop people disbelieving in God, so there’s a lot of convincing being attempted, people seeking each other out to tell them that they cannot or must not hold a certain belief, and so forth. This topic is very different! If I want to go off and believe LLMs can understand and reason and think, you don’t really care; likewise, when you believe that, although it might be producing a novel rephrasing, the information in that rephrasing is merely reproduced from the training corpus, I don’t really care. (Beyond active engagements in discussion threads, anyhow - not trying to imply either of us don’t care about the discussion we’re having right now!)

But in other ways it is very similar to the atheism/theism debate. For a time I used to argue God exists because the idea of God controls and causes very specific behaviors from a great many people; a naive diagram of the situation has this God figure and He’s telling these little people figures to do certain things and they do exactly those things (which is why “pointing out that certain figures are hypocritically not doing those things” feels important and like it might be evidence against), and the atheist argument that you can’t actually detect God’s existence is like erasing the God figure from the diagram: it leaves a conspicuous hole in the diagram, and the hole is shaped like God, and if you were to naively fill in the hole in the diagram to explain what’s causing the people to behave that way, the precise shape of the hole forces you to end up re-drawing God (this argument was very unconvincing to atheists for multiple reasons and I’m not trying to re-litigate it here).

It became clear to me that atheists were not interested in this level of proof, it had to be physical existence; some ‘God-stuff’ that you could grab a handful of and hold up to the light and say “here it is”. This topic is very similar! For me, it is enough that the diagram of an LLM’s capabilities has a hole shaped exactly like “it understands” and I’m happy to fill in the obvious blank; for you, there needs be an actual process of understanding somewhere in the code and the weights that we could hold up to the light and say “here is the understanding” (and that is how you can claim that no matter how good LLMs get they will never be understanding - all of their processes are already fully described and explained, there’s nowhere for a process of understanding to hide).

>> Perhaps you calculator understands arithmetic indeed!

Well, perhaps it does. But that depends on a definition of "understands" that we can all agree covers humans, calculators, and large language models, alike.

And what about cats and dogs and other mammals? Surely whales "understand" something, say, about fishing, even if they can't do arithmetic?

How about insects? Does a cockroach understand anything?

To be honest, I doubt we can have some kid of a priori definition of "understands" capable of convering all of the above. I think we have to wait for a lot longer until we can really say for sure what "understand" even means in humans, in the first place, let alone everything else.

You can say this same thing about any human too. After all, how can you be sure that my mind "understands" the same way your does? If your argument is that we have similar hardware, you must point out what's so special about this biological brain that enables "understanding" and why neural networks don't.

If you ask that question to humans most wouldn't be able to solve it. Even with access to a calculator. Not being able to do it just means it's not good at maths.

I agree there is a connection between ChatGPT, god and atheists but I don't think it's the one you point out. The commonality "general intelligence" and "god" share is there is no agreed definition of either.

Science has forced a steady retreat in the number of things that can be attributed to god. But that shouldn't worry theists, because Godel's incomplete theorem means there are some things we will can never understand in a finite series of steps, so there will always be room for god. God explains the "rest". I suspect the "rest" is infinite, so that leaves a lot of room for god.

Whether LLM's are intelligent or sentient falls into the same category. As intelligent and sentient don't have a formal definition you can always change their definitions, and that's exactly what seems to be happening. A task is put up as "only humans can do this" (like the Turing test, or a captcha) then it's solved, then the goal posts are moved. Again, because of Godel's theorem I can say with certainty this is a game we can continue to play forever.

I debated on the atheists "team" in forums for a decade or two before it dawned I was effectively arguing about how many angles will fit onto the head of a pin. It's a peculiar game. Godel's theorem effectively says there is a no finitely sized rule that can describe a system of infinite states. As we are finite it's possible (in theory) we could one day find a rule that can predict every situation we could see, which means in the "atheists vs theists" game we win every battle because we've explained (or predict even) everything we experience. But the universe is infinite, so while we can explain everything we've experienced we can't explain everything there is. So we can win every battle - but still never disprove god because we can never experience everything. Thus we lose the war.

In other words I lost the "there is no god" debate. But it wasn't because I now think there is no god. It's because I realised I could never prove it, which to me is the same as losing it. The situation is exactly the same with "AI's are intelligent" debate, which makes it's just as pointless.