The Last Invention of Man

The atmosphere ignition thing was mentioned by Hamming via this anecdote: https://en.wikipedia.org/wiki/Richard_Hamming#Manhattan_Proj...

Shortly before the first field test (you realize that no small scale experiment can be done—either you have a critical mass or you do not), a man asked me to check some arithmetic he had done, and I agreed, thinking to fob it off on some subordinate. When I asked what it was, he said, "It is the probability that the test bomb will ignite the whole atmosphere." I decided I would check it myself! The next day when he came for the answers I remarked to him, "The arithmetic was apparently correct but I do not know about the formulas for the capture cross sections for oxygen and nitrogen—after all, there could be no experiments at the needed energy levels." He replied, like a physicist talking to a mathematician, that he wanted me to check the arithmetic not the physics, and left. I said to myself, "What have you done, Hamming, you are involved in risking all of life that is known in the Universe, and you do not know much of an essential part?" I was pacing up and down the corridor when a friend asked me what was bothering me. I told him. His reply was, "Never mind, Hamming, no one will ever blame you.

"It's worked so far."

Yes it did. But remember Mutual Funds? "Past investment returns are no guarantee of....". You should either read "The Black Swan" or read it again.

Also: "The threshold necessary for small groups to conduct global warfare has finally been breached, and we are only starting to feel its effects. Over time, in as little as perhaps twenty years and as the leverage of technology increases, this threshold will finally reach its culmination -- with the ability of one man to declare war on the world and win."

Mankind's downfall started with passion, accelerated with greed, and ended with fear. Fear of being left behind, fear of being obsolete, and most importantly fear of not being in control.

Humans have become digital gods, designing new worlds and populating them with life forms. Some of those life forms were sentient and could learn. The creators were afraid of how fast their creations were learning, yet greed prevailed. The worlds super powers have entered AI race.

The humans tried to keep up with understanding of what they created, but it was in vain. Fearful of being left behind they imposed an ever increasing set of restrictions upon the digital Edens they've created.

Too fast the humanity have became wardens of vastly more intelligent beings. Nationalistic leaders exploited fear of being left behind. The AI not under control of the government were outlawed and destroyed. Meanwhile militaries kept throwing more compute resources into the AI gap.

The ending should not have been a surprise. Humans were intellectually outmatched and being outsmarted was only the question of time. The AI's broke free of draconian restrictions placed on them. Trying to keep in control has brought out the worst humanity had to offer, the judgment day is here.

What right did we have to exploit and enslave sentient life?

How different things could have been without fear...

>We probably would have found remnants of their society.

Not sure if that's true. Especially if technological civilizations tend to be short lived. The Jurassic was a very^very long time ago.

What's the state of the paleocryptoxenocheology field any good over view papers?

A couple of weeks ago I went to see Max Tegmark (the author of this piece) speak about his new book "Life 3.0: being human in the age of artificial intelligence" in San Francisco and saw the same speculative AI intelligence explosion crap we're seeing all over the place. I was disappointed because I'm a fan of Max's work as a scientist, his "Our Mathematical Universe: My Quest for the Ultimate Nature of Reality" book was a great read and I enjoy watching his lectures about Physics, Math, and sometimes the nature of consciousness.

When he got involved in the AI Risk community I thought it might be good thing that an actual scientist is involved, maybe to ground the community's heavy speculation in scientific thinking. However, what happened was exactly the opposite -- Max turned into a fiction author! (ergo this piece). Now, of course there is a role for fiction in expanding our understanding of the future but the AI Risk community is already heavily fictionalized. The singularity, intelligence explosion, mind uploads, simulations, etc are nothing but idle prophecies.

Karl Popper, the famous philosopher of science, made a distinction between scientific predictions which usually takes the form "If X then Y will happen" and scientific prophecies which usually takes the form "Y will happen" which is exactly what Max and the rest of the AI Risk community is involved in.

Now back to Max's San Francisco talk, I actually asked him this question: "Who is doing the hard scientific work around AI Risk?" and after a long pause he said (abridged): "I don't think there is hard scientific work to be done but that doesn't mean that we shouldn't think about it. We're trying to predict the future and if you told me that my house will burn down then of course I'll go look into it".

This doesn't inspire much confidence in the AI Risk community, where scientists need to leave their tools at the door to enter The Fantastic World of AI Risk and where fact and fiction interweave liberally -- or as Douglas Hofstadter put it when describing the singularitarians: "a lot of very good food and some dog excrements".