The End of an Era

The particular problem I have with your take on value is it sets up a future conflict based upon the state that a human has. It's based on a number of presumptions that only a particular set of arrangements of matter, well, matter. And those that matter happen to be watery sacks of meat.

So in your framework you setup a definition of what an external measurement of someones sentience state... Now, someone makes a machine that outputs those states and externally behaves as if it has sentience. What do you then? Upturn the table and get mad because your classification system no longer just contains humans?

Thank you for your reply. I'm not sure I understand your objection perfectly.

> it sets up a future conflict based upon the state that a human has

It's not clear to me it does.

> It's based on a number of presumptions that only a particular set of arrangements of matter, well, matter.

Well, this premise must be true, I think. How could it be otherwise?

> And those that matter happen to be watery sacks of meat.

This I disagree with. My comment contains no such claims that only humans matter.

> So in your framework you setup a definition of what an external measurement of someones sentience state... Now, someone makes a machine that outputs those states and externally behaves as if it has sentience.

So, this is something we need to be very cautious about, I think. First, we don't really know what makes someone sentient. I do not claim to know, and certainly not know with a high degree of certainty (I may have a few incomplete hypothesis).

First, it's clear to me that someone's external behavior does not dictate sentience (positively or negatively). I gave some examples, but I think this can be pretty reliably established. To make things simpler, divide the candidate sentient systems by their "output quantity/quality", from less to more. (1) On the "no output side", it's clear that someone can think a lot and have a wonderful inner life without being able to speak. There are (some medical) cases of people going long periods without speaking by choice or by medical condition. Those people can be sentient just fine, obviously you shouldn't need to speak or "output" anything to be considered having an inner life. (2) On the "human-like output" (or superhuman), the argument becomes more involved. First, clearly just the output itself being indistinguishable from human is not sufficient because it could be for example a simple program that just pastes some verbatim copy of a never seen dialogue. Or very simple chatbots that are quite convincing at appearing human (can fool many judges for a while), but clearly also are not sentient. Even for a for a system that genuinely resembles a real human even with under scrutiny, (with all sorts of similar abilities across all fields), consider the case that this system could be simply one astronomically large (unrealistic, I know) lookup table. Basically, it simply works inside by fetching an indexed piece of data and returns it. This is possible, although this machine would need to be astronomical in scale (so it's more of a thought experiment). It shows to me, convincingly, the actual "internal" or "microscale" even implementation of your thoughts matter critically for sentience. I presume, for example, that even if some small portion of your mental activity (i.e. a kind of "subcircuit" in your brain) were replaced by something like a lookup table, you would lose part of the experience of that mental activity. This is clear when the content of your mind differs when you're trying a (kind of) problem for the first time and when you're solving one for the umpteenth time. The latter (possibly, not necessarily) feels more boring, also normally more automatic with less thinking required as you've built habitual responses.

> What do you then? Upturn the table and get mad because your classification system no longer just contains humans?

I would do no such thing. I would welcome any other kinds of sentience. But it would be important to me to carefully establish and understand what kind of systems are sentient, indeed in large part to protect them.

---

My main point is that you kind of need to start with sentient minds to understand what sentience is (and of course, not only that, but sentient minds that are aware of their sentience and willing/keen to study it). If you start without sentience, it doesn't seem like this particular concept would form a necessary basis for a theory of action or motivation. Like you can't prove from elementary mathematical premises that sentience is important, I don't think. We need to start from some knowledge that derives from first-hand experience of being an actual sentient being, and expand, solidify or formalize this knowledge to be able to understand it with scientific precision (or anything close to it).

>some knowledge that derives from first-hand experience of being an actual sentient being

This is where things kind of fall apart. We all agree (hopefully) that humans are sentient. The question is, "what else is sentient". Are all animals sentient? If so that leaves a very, very wide range on what sentience is. If not you have to operate your scientific scalpel with a very high level of subjectivity on what is and isn't counted. At some point in the past there was no sentience, now there is. Was it a click and we have it, or was it a slow awakening to the world bit by bit (this is my take). It would also quite likely miss systems that could operate at a higher level of sentience than humans, for example some types of hive minds.

My take on it is that any sufficiently complex system with memory, the ability to perceive outside information, difference that information from what it's doing versus what's happening outside of its actions, and learn from that behavior is at least somewhat sentiment.

This leads me to think that LLMs are 'kind of' proto-sentient. They are not persistent, their short term memory is tiny, and they don't self learn. Once we get the persistent and self learning parts working this really starts to check a lot of the boxes that we consider sentient. Of course by the time we get that working it seems likely we'll have much bigger problems in dealing with AI and the entire sentience thing might not be the first thing on our plates.

> This is where things kind of fall apart. We all agree (hopefully) that humans are sentient. The question is, "what else is sentient". Are all animals sentient? If so that leaves a very, very wide range on what sentience is. If not you have to operate your scientific scalpel with a very high level of subjectivity on what is and isn't counted. At some point in the past there was no sentience, now there is. Was it a click and we have it, or was it a slow awakening to the world bit by bit (this is my take).

I believe pretty much the same thing. It probably isn't all or nothing.

But for me personally I'm really interested not only in what it is (although I'm very much interested in that), but because the problem is so thorny (as you mention subjective), I am first centrally interested in how (or if) we can actually know it reliably. So subjectivity is part of reality, as the mind is part of reality. In principle, it seems we should be able to understand quite a bit about subjectivity because of this, because it is part of reality. For example, the mind clearly obeys causality and all other principles (that I know of) we observe (elsewhere) in reality.

So I've been studying this along side with ethics, as the two are deeply interrelated, and I think ethics is an important field to develop, and to develop additional certain around (many people find difficulty in finding meaning in life). By ethics I mean what we should do, what is good, and from that also of course how we should behave toward each other.

You can read some of my studies here: https://www.lesswrong.com/posts/Dxco4XDttx5cNWjfj/how-to-dis...

and here: https://old.reddit.com/r/slatestarcodex/comments/1iv1x1m/the...

I believe in a very open science way of working, feel free to contribute :)

That said, I have no formal philosophical training (although I did study so philosophers), but I believe for philosophy as for math and science the soundness of the arguments (as well as consistency with reality/experimental verification) are what is fundamental.

> was it a slow awakening to the world bit by bit (this is my take)

Yes, I usually have been facing this kind of problems using some tools inspired by math and logic. For example, take some supporting arguments (1) it is known that humans undergo significant changes throughout our lives, we gain neurons and neural connection in childhood, and lose those throughout our lives (although I believe the brain as a whole is very dynamic throughout). There didn't seem to be any turn on/off of sentience in my own life; (2) Even my earliest memories point to a gradual awakening; (3) In my discussion with Dagon, he brought I think a central point: probably sentience isn't a single phenomenon or something you turn on/off. It's probably more a whole array of inner sensations (that may compose greater structures, longer lasting sensations), kind of like a zoo of experiences. It's us that desperately want to be a single, or singular thing, which I believe is related to our instincts and notion of ego. So the self or ego seems like it is not a fundamental thing, but simply a way we understand or organize our society, and I do think it's important in practice. I think we're more like a collection of interrelated internal phenomena, which can probably significantly vary in intensity, duration, how good they are (this is a key and tricky point I believe!), etc.. This might be analogous to fields and particles in physics. As far as we currently know, it seems fields are arguably more fundamental than particles (or rather a possible distinct and complete interpretation), and fields are extended and not discrete phenomena. Particles are useful approximations or abstractions of field behavior in this sense. In closure, indeed sentience does not seem like anything like an on-an-off, 0/1 phenomenon, as explained in the cases above.

I really want to know how much can we know about sentience. If my hypotheses above are true, then I think we can establish some elementary facts only about sentience, like you mentioned it seems clear to me as well you need some kind of information processing system, the volume of information probably matters, and its internal structure probably matters as well. Consider that in any object at a temperature above 0K, there is an enormous flux of information corresponding to interaction between particles, essentially thermal flow or local kinetic information. Clearly any piece of matter is not sentient. So clearly what information is being exchanged and how, how it is processed and transformed matters. I think a useful principle in this regard is simply that the information should (as a sufficient, and maybe almost necessary condition) represent certain things and form networks of representation or meaning. Basically, sentience (which gives meaning to life) seems pretty much built on meaning: in the sense of representation and interrelation of abstract concepts, for example, like the state of your body, known persons and ideas, places, and so on (some of this meaning seems quite instinctive and non-verbal, I do not believe human language is a total necessity for those experiences to manifest, I think there are proto-linguistic representations in our minds; so I do believe animals probably do have varying degrees of sentience; indeed many of those meaningful representations many times are just bodily signals like pain, relief (like when you drink water after being thirsty), taste from food, or abstract feelings from music, and so on).

So as we require a more and more complete (e.g. finding more specific necessary conditions for them to manifest, and also more detailed sufficient conditions) conditions for those sentience phenomena, we will I think meet a vast array, a huge number of distinctive phenomena, and those phenomena manifest in different character, intensity and frequency depending on the individual (like comparing what a human experiences with some animal of other species). I think that will find close associations with the study of art. Like we know there is more energetic or more relaxed music, or more melancholic or more upbeat, and those feel different. We already know a whole lot about sentience :)

Now we do come back to the question of whether we can tell say an LLM manifests those phenomena internally. Beside everything I said, I think it's a matter of studying its structure and seeing comparatively how much it transforms and analyzes information (with a certain originality, perhaps, i.e. non-memorization) versus how much its answers are simply kind of rote and non-transformative (basically a high degree of memorization) (like the lookup table example I gave previously). Those are all observations built on our first hand experience of being sentient beings, and couldn't be otherwise -- the "laws of the mind" are not provable in elementary systems of logic (somewhat like the laws of physics are not either). Logic is able to tell only which of a number of candidate theories is internally consistent (note logical consistency itself has an experimental motivation from observing nature! (e.g. logic is related to causality in nature)), but it cannot tell which of candidate consistent theories is the case. For example, someone can claim sentience simply does not exist. There is nothing logically inconsistent in that theory by itself, but its inconsistent with our own experience of the world (I think, therefore I am). I've seen some pretty wild claims on the internet :) which I don't believe are true -- you can find propositions that the meaning of life is this or that (for example, to have power or wealth) -- I reject those propositions in a similar manner to stating "I think, therefore I am" -- sentience is evidently true from personal experience, and it being true, and us essentially being "made out of" sentience or sentient phenomena, all that matters must concern that, and not any external objects like wealth or power. It's clear to me wealth does not automatically determine the character of your sentience, and neither does power.

Another central interest of studying sentience is understanding whether some experience (or set thereof) is good or bad -- for the lack of being aware of a better term, I call this valence (i.e. how much value it has). That is important of course to us, because we don't want to just live, we want somehow to have good lives. This is something else that is experimentally true, I believe: when you suffer or feel intense pain, for example, you can verify that indeed there are bad feelings; and likewise when you have gone through good moments, afterwards you become convinced of their existence. This is kind of tricky because I don't think what we desire is necessarily good, or things we don't desire necessarily bad (i.e. it seems motivation, which is a lot driven by instinct, is distinct from genuine valence). And in general this self-knowing process has many important personal limitations like our limited memories and limited capacity to compare different feelings.

---

Finally, it is not clear to me this project (for example, of being able to have some rough scientific measure of how intense and frequent are sentient phenomena (or qualia) of an LLM versus a typical or given human -- say e.g. 50,000x more or less intense) is at all feasible, given the limitations of humanity and technology, it's not clear we can do more than handwave "Something, something information processing, something something meaning!" :) But I do think it's worthwhile to try and see what we can find.

It's a big part of why I'm strongly opposed to common-law systems (as opposed to civil law). All the case law, all of it becomes, well, law. And then you get the whole "legislating from the bench" critique, which, while valid critique, is only a big problem because it's a common-law system.

Yes, I realize many (most?) legal systems are actually hybrid, but they do tend to err pretty strongly into one direction or another.