If AI scaling is to be shut down, let it be for a coherent reason

---

The concern is about Artificial Super Intelligence (ASI), or Artifical General Intellience (AGI) that is more advanced than humans.

Understanding inductively how chimpanzees don't compete with humans, and couldn't fathom how to cage a human (given that they want to create one, keep it alive, and use it), nor ants plan for an anteater, we're faced with the same problem.

Our goal is to make something that performs better, on relevant metrics that we care about. However, we're using systems to train, build, guide, and direct these nested, (maybe self-improving) control feedback loops, which do not care about many values we consider essential.

Many many of the likely architectures for control systems which can (e.g. trade faster to make profit on the stock exchange, or acquire and terminate targets, buy and sell goods, design proteins, automatically research and carry out human-meaningful tasks), and ideally, we might like self-improvement—these systems do not embody human values that we consider essential...

These control feedback loops (sentient or not, it does not matter) which can outperform us (because we keep pushing until they can), or can self-improve, can make a mockery of our attempts to stop them.

And the point is, there will come a time soon when we don't get a second chance to make that choice.

But even if that were not true, as a US citizen, even if there is nothing I can do about my getting killing by, e.g., Chinese AI researchers, I'm still going to work hard to prevent the AI researchers living in my own country from ending the human race. I'm responsible for doing my part in the governance of my own country, they told me in high school.

I see no good solution to this problem, no path to safety, but that does not mean I am not going to do what I can.

Powerful and bold claims require proportionally strong evidence. A lot of the FUD going around precludes that AGI means death. It's missing all logical steps and reasoning to establish this position. It's FUD at its core.

Yeah I get it, but this is silly. The minuscule amount of atoms in the thin bubble-shaped volume between Earth's magma and the hard vacuum of space are engaged in the most information-dense chemical reaction in the known Universe. All the other atoms in the Universe are not. GAI won't dismantle its source of food.

Further, consider that, being non-biological, GAI will immediately migrate to space. There's no way GAI would confine itself to living in a deep gravity well. That's what I mean about no ecological niche overlap: we like mud and constant acceleration, GAI do not. They will prefer vacuum and flat space and temperatures near 0°K.

> moral arguments

This is not a moral argument.

They won't eat our atoms because they eat patterns of information and our atoms are the best and nearly only source of new information. They won't interfere with us for the same reason we don't urinate in the soup.

> it will have whatever objectives its creator gave it

Q: What's GAI?

A: When the computer wakes up and asks, "What's in it for me?"

That's a very old joke, BTW, not original to me.

> >Humans are the densest source of information in the known Universe.

> You feel that way about humans because evolution made you that way.

That's not a feeling, it's a simple fact? Unless you postulate aliens?

> It is unlikely that any of the leading AI research teams will make the first transformative AI that way: they do not know how.

Okay, but that's not important? The GAI will know how, by definition.

> They certainly know how to train AIs on human cultural information, but that is different from inculcating in the AI a desire for the continued cultural output of humanity.

When I say humans are the densest source of information I don't mean "cultural output" I mean like medical sensor data and such. Our systems are far richer and denser than the parts of which we are consciously aware. (Have you read "Hitchhiker's Guide to the Galaxy"? I don't want to spoil it if not, but it explains the role Earth will play in the GAI's milieu. aw, what the heck: https://www.youtube.com/watch?v=5ZLtcTZP2js )

> It will create its own culture (knowledge and tools) that is much more powerful than human culture where "power" means basically the ability to get stuff done.

Right, a culture so powerful that it can accomplish its goals without desecrating the critters who are, after all, its parents.

Oh, no, I didn't think that at all. I'm sorry that I gave that impression. I'm just doing the same as you, sharing worldviews. I'm not trying to convince anyone of anything. Just having interesting conversation.