Modern-Day Oracles or Bullshit Machines? How to thrive in a ChatGPT world

I think I get what you're saying there - they are not conscious in the same way that humans are - but "consciousness" is a highly-debated term without a precise definition, and correspondingly philosophers have no consensus on whether machines in general are capable of it. Here is one of my favorite resources on that, the Stanford Encyclopedia of Philosophy's page on the Chinese Room Argument:

https://plato.stanford.edu/entries/chinese-room/

Things that appear conscious, or that appear to understand a language, are very hard to distinguish from things that actually are those respective things.

Again, I think I get the intended point - some people interact with ChatGPT and "feel" there is another person on the other side, someone that experiences the world like them. There isn't. That is good to point out. But that doesn't mean machines in general and LLMs specifically can't be conscious in some other manner, just like insects aren't conscious like us, but might be in their own way.

Overall I think the general claim "LLMs aren't conscious" is debatable on a philosophical level, so I'd suggest either defining things more concretely or leaving it out.

Jet planes are so far removed mechanically from a bird that the idea they fly is not even remotely worth considering.

I probably don't.

> For one, it's not even internally consistent -- the people who built telescopes and satellites didn't "see" the moons, either. They got back a bunch of electrical signals and interpreted it to mean something.

I'm not trying to go 100% reductionist here; I thought the point was clear. I was locking on the distinction between "learn from experience" vs. "learn from reading about it", and corresponding distinction of "test by predicting X" vs. "test by predicting other peoples' reactions to statements about X" - because that's the distinction TFA assumes we're on the "left side" of, LLMs on the "right", and I'm saying humans are actually on the same side as LLMs.

> This worldview essentially boils down to the old "brain in a jar" which is fun to think about at 3am when you're 21 and stoned, but it's not otherwise useful so we discard it.

Wait, what's wrong with this view? Wasn't exactly refuted in any way, despite proclamations by the more "embodied cognition" folks, whose beliefs are to me just a religion trying to retroactively fit to modern science to counter diminishing role of human soul at the center of it.

> I'm always a bit shocked and disappointed with people when they go "well, you learn it on a test and that's how you know" because ...no, no that's not at all how it works. The most essential part of learning is knowing how we know and knowing how certain we are in that conclusion.

My argument isn't simply "learn for the test and it's fine". I was myself the person who refused to learn "for the test" - but that doesn't change the fact that, in 99% of the cases, what I was doing is anticipating reaction of people (imaginary or otherwise) who hold accurate beliefs around the world, because it's not like I was able to test any of it empirically myself. And no, internal belief consistency is still text land, not hard empirical evidence land.

My point is to highlight that, for most of what we call today knowledge, which isn't tied to directly experiencing a phenomena in question, we're not learning in ways fundamentally different to what LLMs are doing. This isn't to say here that LLMs are learning it well or understanding it (for whatever one means by "understanding") - just that the whole line of arguing that "LLMs only learn from statistical patterns in text, unlike us, therefore can't understand" is wrong because 1) statistical patterns in text contain contain that knowledge, and 2) it's what we're learning it from as well.

I would maybe accept debates about whether for example ChatGPT (the whole system that stores old conversations and sends the history along with the current user entry) is conscious - but just the model? Isn't that like saying the human brain (just the organ lying on a table) is conscious?