Life is more than an engineering problem

“Feeling” is disconnected from reality, it’s whatever you perceive it as. Like morality, you can’t disprove someone’s definition of feeling, you can only disagree with it.

If scientists invent a way to measure “feeling” that states 20% of people don’t feel, including those otherwise indistinguishable from feeling ones, most people would disagree with the measurement. Similarly, most people would disagree that a printer that prints “baby don’t hurt me” is truly in pain.

These discussions seem to me to still get hung up on the classical sci-fi view of an AI, even talking about Companion here, of some single identifiable discrete entity that can even potentially be the locus of things like rights and feelings.

What is ChatGPT? Ollama? DeepSeek-R1? They're software. Software is a file. It's a sequence of bytes that can be loaded into memory, with the code portion pulled into a processor to tell it what to do. Between instructions, the operating system it runs on context switches it out back to memory, possibly to disk. Possibly it may crash in the middle of an instruction, but if the prior state was stored off somewhere, it can be recovered.

When you interact through a web API, what are you actually interacting with? There may be thousands of servers striped across the planet constantly being brought offline and online for maintenance, upgrades, A/B tests, hardware decommissioning. The fact that the context window and chat history is stored out of band from the software itself provides an illusion that you're talking to some continually existing individual thing, but you're not. Every individual request may be served by a separate ephemeral process that exists long enough to serve that request and then never exists again.

What is doing the "feeling" here? The processor? Whole server? The collection? The entire Internet? When is it feeling? In the 3 out of 30000 time slices per microsecond that the instruction executing is one pulled from ChatGPT and not the 190 other processes running at the same time that weren't created by machine learning and don't produce output that a human would look at and might think a human produced it?

I'll admit that humans are also pretty mysterious if you reduce us to the unit of computation and most of what goes on in the body and brain has nothing to do with either feeling or cognition, but we know at least there is some qualitative, categorical difference at the structural level between us and sponges. We didn't just get a software upgrade. A GPU running ChatGPT, on the other hand, is exactly the same as a GPU running Minecraft. Why would a fMRI looking at one versus the other see a difference? It's executing the same instructions, possibly even acting on virtually if not totally identical byte streams, and it's only at a higher-level step of encoding that an output device interprets one as rasters and one as characters. You could obfuscate the code the way malware does to hide itself, totally changing the magnetic signature, but produce exactly the same output.

Consider where that leads as a thought experiment. Remove the text encodings from all of the computers involved, or just remove all input validation and feed ChatGPT a stream of random bytes. It'll still do the same thing, but it will produce garbage that means nothing. Would you still recognize it as an intelligent, thinking, feeling thing? If a human suffers some injury to eyes and ears, or is sent to a sensory deprivation chamber, we would say yes, they are still a thinking, feeling, intelligent creature. Our ability to produce sound waves that encode information intelligible to others is an important characteristic, but it's not a necessary characteristic. It doesn't define us. In a vacuum as the last person alive with no way to speak and no one to speak to, we'd still be human. In a vacuum as the last server alive with no humans left, ChatGPT would be dirty memory pages never getting used and eventually being written out to disk by its operating system as the server it had been running on performs automated maintenance functions until it hits a scheduled shutdown, runs out of power, or gets thermally throttled by its BIOS because the data center is no longer being actively cooled.

I think Ted Chiang is doing us a service here. Behavioral equivalence with respect to the production of digitally-encoded information is not equivalence. These things are not like us.

OK, so I asked ChatGPT how to plant potatoes on Mars...

=================================================

Planting potatoes on Mars would be a pretty fascinating (and challenging) task! While Mars has conditions that make growing traditional crops difficult—like low temperatures, low atmospheric pressure, and a lack of oxygen—scientists have been experimenting with ways to grow plants in Martian-like conditions. Here’s an overview of the process:

1. Create a Controlled Environment:

Temperature: Mars’ average temperature is about -80°F (-60°C), far too cold for potatoes to grow. You’d need to create a greenhouse-like environment, potentially using materials like glass or transparent plastics, to keep the temperature warmer.

Atmosphere: Mars has only about 1% of Earth's atmospheric pressure, which is too low to support plant life. A sealed greenhouse would be required to provide a breathable atmosphere with adequate pressure and oxygen levels.

Light: Mars has less sunlight due to its distance from the Sun. You would need supplemental artificial light (perhaps LEDs) to mimic Earth’s daylight cycle for the plants.

2. Soil: Mars has soil, but it’s not exactly like Earth’s. It lacks organic material and has toxic elements like perchlorates that would harm plant growth. Scientists would need to either:

Modify Martian soil by adding organic material (like compost) and removing or neutralizing toxic chemicals.

Hydroponics: Grow potatoes without soil, using a nutrient-rich water solution instead.

Obviously I meant that the content of the results can be inaccurate, and I assume you weren't actually confused about that.

Ok, the first example I found was when I was trying to find a way to write a rust proc macro that recursively processes functions or modules and re-writes arithmetic expressions. The best way to do this, it turns out, is with `VisitMut` or `fold`. I cannot find any results discussing these approaches with google, but ChatGPT (4) suggested it within the first couple refinements of a query.

Another recent example from my history: "can you implement Future twice for a rust struct, with different Output types"