They’re made out of weights

So subjective experience is subjective experience? Do rocks have subjective experience? The main goal of any definition is to say clearly what is doesn't cover. It is not enough to say what is subjective experience, you should say it in a way that excludes everything else. Like what is objective experience? How it differs from subjective one?

Do LLM experience objectively or not experience at all? How can you say?

> But you cannot say the opposite either, so saying 'machines could think' base on a false assumption

I can say that they could think. The thought process implies a measurable product. Yes, there are similar situation with it, it can be hard to say sometimes if we observe a product of a thought or something else. But science has some success with this, like claiming that bees can think and solve problems.

> That's a very narrow understanding of what language is, what linguists do/research, and the contributions made in the field. Linguists are (since already 2 o 3 decades) focusing more and more on psychological/cognitive matters.

I'm telling you, psychologists (who specialize on mind research) do not know what consciousness is and they do not have a definition of a subjective experience (well, if we treat your proposed definition as a valid definition, then we should say they have plenty). And my claim still stands: until you heard about new science "Subjectology" you can be sure that no one knows what subjective experience is. Including linguists.

> the topics you mention here are already settled in the philosophy space

You shouldn't believe that anything can be settled in the philosophy space. Philosophers can think they have settled things, but until science agreed and started empirical studies, it is just philosophers believing that they settled things.

> Exactly because of this. And is this what I am talking about...

Exactly because of this I'm vary of any "settled things". You see, people have all the reasons for motivated reasoning. People are historically very wary of any attempts to extend the group of beings covered by "human rights" or whatever it was actual at their time. People would fight to the death to not include another group of beings to the list of "sentient", "conscious" or what it is today is euphemism to "morally equal to a human". And I do not trust anyone to overcome this deep psychological bias. I do not trust myself, I do not trust philosophers, I do not trust linguists, I do not trust neurologists and psychologists. Well, psychologists are probably the most prepared to it, but I do not trust them still.

Just think about it, lets suppose someone proved beyond any doubt that LLM has conscious experience. What happens then? Try to imagine that counterfactual world. It will be a very troubled world, won't it? Any sentient human feels it, they feel the level of push back they would receive if they claimed that LLM is conscious. So any sentient person unconsciously feel the urge to join the side claiming that LLMs are not conscious: it is safe to argue they are not, you can publish papers on it and you won't face a strong condemnation from angry scientists, politicians, large corporations hoping to make a shitton of money with LLMs, various religious groups, etc. Our world just have no place for a sentient LLMs. If they are sentient we have to either ban them, or to do sweeping changes to make room for LLMs with human rights.

No philosopher/linguist/neuroscientist/psychologist is safe from this unconscious bias, and they do not speak math, they say things like your definition of subjective experience above. Things that can mean anything if you try hard enough. It means you can't just take logic and check their reasoning, you have to _feel_ that they are right, but your feelings a subject to your psychological biases too. You can trust your subjective experiences because they are subjective.

If philosophy "settled things" then I even more wary of it. It has even more biases to keep status quo.

Wanna me to draw you a picture how the philosopher opinion would evolve if it turns out that LLM are conscious? History shows us how these things unfold. It won't happen overnight, people working with LLMs will notice some patterns and they will ask new questions and slightly modified versions of "already settled questions". The latter will be forced into "already settled answers" (Thomas Kuhn, normal science phase). The former might be answered in the current paradigm (and become settled) or just thrown away as "meaningless" questions. It will be a long process: people noticing things, world class thinkers forcing them into the paradigm. With each step it will become more and more ridiculous, until everyone will see that paradigm doesn't hold. After that politics will take over and it will be defending "traditional paradigm"... well, I can't predict the next phase, but it doesn't matter.

What does matter is the alternate Universe where LLMs are sentient would look to you just like ours. Matters settled already, all questions have answers. Well, some don't, but those are nonsensical questions. etc. etc.

None of that helps you understand how exactly LLMs do what they do. Because it describes an interface, not a mechanism.

The inner mechanisms of an LLM are more learned than designed. We know what an LLM does on a low level, but going from that to understanding how they work is like trying to understand how a web browser works by looking at netlists of a CPU. Low level understanding does not grant you high level understanding for free.

But ignoring all of that lets you cling to a very comforting "we understand LLMs because we made them". Ha ha. As if.

> And we also know that human beings do not hold 'internal representations' like any AI system needs to.

Bold fucking claim. Got a source on that?

Because neurobiology has been trying to crack neural representations - the very internal representations brains use - for as long as it existed, and with some success. Both reading and injecting internal representations into the brain is possible now, in narrow cases. The specifics vary region to region, but sparse population coding is a true staple. Today's SOTA for wrangling this mess is ML decoders, and not by a coincidence.

My original comment was that it seems (and it is actually documented in the books I referenced) that the AI research space builds its claims on assumptions, not on facts, and that those assumptions are flawed. So a nice discussion, to begin with, would consider:

1) why I make the claim that the AI research space builds its claims on assumptions instead of facts, why we could say that there are actually no assumptions but facts, or why the assumptions are correct.

2) instead of strictly and directly dismissing readings on philosophy, I would expect intelligent and curious people to embrace new references. Particularly if those references are highly regarded and a solid contribution during the last 120 years

Regarding point 1), I can barely count a single comment in this thread that tries to engage in the idea of the assumptions (except for some comments that agree with the premise).

Then regarding point 2), I can barely count research papers, books or contributions in the space of AI research that references (either to built upon or dismiss) philosophy that is pertinent to AI, pertinent to philosophy of technique or cognitive linguistics. This is strange. It looks like if the space revived during the 2000s with the invention of neural nets (RL, GAN, etc), and then became isolated from contributions about human intelligence, even though it continually tries to explain intelligence in its own terms.

The reference to What Computers Can't Still do is precisely relevant because it narrates exactly this same discussion (false assumptions, claims built upon assumptions instead of facts, dismissal of evidence from psychology, dismissal of frameworks from philosophy, fallacies about progress), but it was written in 1972. Still, you read the book today, and it is totally relevant.

Now, regarding your comments:

> Do LLM experience objectively or not experience at all? How can you say?

The world cannot be experienced 'objectively'. If they experience the world most probably you won't notice. Given that the only way of interacting with an LLM is through a process initiated solely by a human actor, it would be difficult to assess whether an LLM experiences anything at all.

> I can say that they could think. The thought process implies a measurable product. Yes, there are similar situation with it, it can be hard to say sometimes if we observe a product of a thought or something else. But science has some success with this, like claiming that bees can think and solve problems.

The moment you say 'they could think', that implies an assumption about the actual possibility of thinking as a process that can be modeled and executed by a machine. There is, as far as I know, no current evidence that human beings process information the same way a computer does it, nor that though processes necessarily imply a measurable outcome.

> I'm telling you, psychologists (who specialize on mind research) do not know what consciousness is and they do not have a definition of a subjective experience (well, if we treat your proposed definition as a valid definition, then we should say they have plenty). And my claim still stands: until you heard about new science "Subjectology" you can be sure that no one knows what subjective experience is. Including linguists.

Psychology is a very broad field with lights and shadows through its short history. Here, and in my original comment, I am not talking about linguists in general, but specifically about cognitive linguistics. The contributions made be the field are significant and mostly lacking in AI research (for example, the idea of embodiment, the rebuttal of generative grammars, prototype theory, frame semantics, among others). What you mention as 'subjectology', would be just psychology. Foucault explains more or less clearly why this cannot be a science, and that's just fine (in The Order of Things).

> You shouldn't believe that anything can be settled in the philosophy space. Philosophers can think they have settled things, but until science agreed and started empirical studies, it is just philosophers believing that they settled things.

Well... certainly nothing can be 'settled' (not even in science, btw), but my point is: there is already enough convincing arguments in the field of philosophy so as to say that current LLM systems do not posses agency or experience, and that they do not behave like us.

Again, read the sources, what are you people afraid of? Just read the sources, and then engage in the conversation.

I can probably to clarify it.

1. I don't want to end in a situation when AI deserves human rights but I deny it. There are moral reasons for that, and they are important.

2. I employ a systemic view. Not just arguments for consciousness or against it. I look at the people generating these arguments, how their minds work? I look at social institutions while they try to find some consensus. I'm very interested in their inner processes of generating truths. IN particular I'm interested in their failings and how they can generate untruths instead.

3. To understand a system I rely on historic data. How people and social institutions (including science) dealt with similar questions before.

The issue is, that the problem of LLM agency has potentially extremely wide implications, I expect people to be afraid of them, I expect social institutions to be afraid of them, so I cannot trust science in this regard like I trust it when it talks about physics or biochemistry.

> 2) instead of strictly and directly dismissing readings on philosophy

I think I have a good idea what philosophy thinks now, and it doesn't seem convincing. It looks like a normal philosophy, not like an established science, so you should take it with a grain of salt.

> The world cannot be experienced 'objectively'.

So why we call it subjective experience then? Probably it is irrelevant, and the reasons are purely historic... or maybe not. How about the idea of computers experiencing things, just not "subjectively" but rather "digitally"? Or choose any other adjective you like. You are arguing against assumptions, but why you just accept the idea of experience with the assumption that human way to experience things is the only possible way?

> I can barely count research papers, books or contributions in the space of AI research that references (either to built upon or dismiss) philosophy that is pertinent to AI, pertinent to philosophy of technique or cognitive linguistics. This is strange.

I believe it is an expected outcome. AI is evolving fast, there are plenty of things to research without establishing connections with other branches of science. No sane AI researcher would stop researching AI to get PhD in linguistics to build a bridge between AI research and linguistics. Probably in an ideal world this shouldn't happen, maybe it is short-sighted behavior of a system, but it is just how things work in our real world.

BTW it is a good example of what happens with all the philosophy when shit hits the fan. When possibility of empirical studies arrives, no one bothers themselves with philosophy of things.

> I am not talking about linguists in general, but specifically about cognitive linguistics.

I studied psychology, I've read some linguists (cognitive ones, because they are in an adjacent field), and you see, I don't have trust in either. They do their research, they find some interesting facts and devise interesting theories, but it is all looks more like a chemistry in the first half of a XIX century, than a chemistry after periodic table was created. They can't find their building blocks to create a sound theory.

> The moment you say 'they could think', that implies an assumption about the actual possibility of thinking as a process that can be modeled and executed by a machine.

No, I'm not implying "modeling a thinking process". We don't know what thinking process is. What we observe in our minds is not thinking by itself, it is some kind of a mirror process in our consciousness. The real thinking is hidden from us, but it creates echoes in our consciousness we can observe. If we don't know how thinking works, we can't model it. BTW the reverse is also true: if we can't model thinking, we don't know how it works.

I'm defining thinking more in terms of a problem solving ability. Like psychologists do. Science still doesn't have a good enough definition for thinking, but it has some definitions that a) operational; b) good enough for some limited tasks. "Operational" means that they are defined in terms how to measure what you define, not in terms of modeling some process.

> there is already enough convincing arguments in the field of philosophy so as to say that current LLM systems do not posses agency or experience, and that they do not behave like us.

Well, I don't argue that current LLM systems do not possess agency or experience, I argue that we should not trust philosophy to be the first who claims that LLM systems got agency, if they really got it. There is a possibility that they will fight to the death against it even if it is true. You see, until philosophy methods successfully proved that something is conscious despite it was deemed unconscious before, we can't really know that their methods really work. Maybe they work, or maybe they just mirror our biases and heuristics.

> Again, read the sources, what are you people afraid of? Just read the sources, and then engage in the conversation.

I hadn't read books you mentioned, but in your words I see nothing that can hint that those books have something I don't considered already. So maybe I'll read them in a future, but I wouldn't postpone my engagement in the conversation till I read them.

2 is a problem for any theory of consciousness.

Regarding 3, I'm not certain "complex things have complex consciousness" is assumed, at least not for all definitions of complex. A crystal might be very complex from a structural standpoint but simple from a temporal evolution standpoint. I don't think there's a unified panpsychist perspective there. From my perspective, it's more a parsimonious rejection of materialist emergence hypotheses than a definitive statement of knowledge.