The Myth of AI – Jaron Lanier

You comment is already forecasting, overvaluing, and then presently justifying yourself using guesses based on a technology you likely don't fully understand.

It is, in some sense, a very simple problem. When and if we advance to the point that AI algorithms are themselves capable of writing AI algorithms, what happens then? Quantify (albeit vaguely) the rate at which AI algorithms can improve AI algorithms. If that number turns out to be roughly as slow as human beings, there's no significant problem. If it turns out that number is greatly larger than human beings, that's a problem.

It is not particularly clear than the number is large; hard problem is hard. No sarcasm; arguably "general intelligence" is the hardest possibly engineering problem. Adding a bit more intelligence to the problem when it will already be getting worked on by a lot of humans may not move the needle much to speak of at all. But it is not particularly clear that that rate is small either, though; humans are smart, yet at the same time, really dumb in a lot of ways. We have terrible working memory (7 +/- 2 items is absurdly small). We have lots of irrational biases. We have lots of things to do in our lives that are not "thinking" (eating, sleeping, etc.), and the vast majority of our brain is dedicated to those problems, not rational thought. We are terrible at manipulating vast symbol systems without taking immense shortcuts, which inevitably completely color our manipulations. What happens when something that lacks those restrictions is turned loose on the problem of writing AI algorithms?

Heck if I know. I'm a human too.

I honestly think that those people who are utterly convinced the rate must necessarily be slow are just as wrong as those who are utterly convinced it must necessarily be fast. We really don't know, but it is hardly an invalid concern.

So far, AI algorithms have not written very many AI algorithms. I've seen some toys where one AI algorithm is hooked up to tune another one, but I'm not sure if any of them have ever been practically useful. (All I know is all the ones I've personally seen have amounted to toys.) And the system as a whole is generally constrained by what the final AI algorithm can output anyhow; if the domain and range of the AI function are immutably fixed, there's not much the AI-meta-system can do to "escape" and do anything terribly nasty.

I do think we're still a ways away from this being an issue. AI is currently still very much constrained to problems a great deal simpler than "writing AI algorithms", which is right up there with the hardest possible things that human intelligences are currently capable of, and that only to a rare few highly trained and highly talented individuals. We're easily decades away from this problem. But when the day comes, well, I wouldn't bet on the self-improving AI experiencing multiple orders-of-magnitude improvement in mere minutes, but I'd sure hate to bet the future of the species that it's not possible. It is not unreasonable to be concerned that the knee could be very steep. We'll know more as we get closer.

With regard to "vestigial monotheism": I think what you're describing is just a property of discourses in general, in that they're all built around bodies of knowledge which have fixed points [assumptions/truths/whatever], and that participants act in such a way as to keep those things stable.

Definitely seen things like this before, very poignant. Not to knock on Kurzweil and friends too hard [lots of interesting stuff in both spaces obviously], but I've had this vibe from talking with self-described Singularitarians as well (there's some overlap between those circles).

I did a poor job of providing examples of religious behavior, but as history shows people can believe and behave in a religious manner about most anything. Including self-driving cars.

I took a comparative world religion a long time ago; I can't easily find the handy behavior list we had to memorize. We were taught something vaguely Mandaville-ian, I don't know if that outlook is still current and cool, but at least you can find Mandaville's opinion in the wikipedia. The "aspects" section of the wikipedia religion article would be a good start.

Shared faith based belief in the inevitability of some rather objectively unrealistic future outcome, in this case "We're all gonna use self driving cars", critically based solely on faith. A mythology, using the "a story that is important for the group whether or not it is objectively or provably true" wiki definition, where progress has always happened in the past and therefore will always happen in the future, when reality is progress only comes from obtaining and using energy, and our petroleum is running out while our population expands, so you do the math on the realistic future trajectory of progress, furthermore there's no question in the religion about the narrative that progress necessarily automatically equals self driving cars, instead of "busses and trains and subways" or not commuting at all or whatever. There's a fair amount of superstition involved aka everything the general public or journalists say about anything technical, legal, or economic WRT the topic of self driving cars, they're just making unreasoned noises at each other. Its always accompanied by extensive and exhaustive ethical baggage, in this case about preferred lifestyles, the value of work, urban planning, as if the self driving car is actually important relative to VPN deployment or subsidized public transit or star trek style transporters, yet despite its irrelevance there's a great pile of associated ethical baggage and related extreme judgment. There's a strong self policing society, where certain aspects of the religion are strongly amplified via endless ceremonial repetition, and the not-devout-enough are tossed out or punished via ridicule or otherwise.

Note that you can discuss self driving cars in non-religious ways, much as you can scientifically study aspects of a religion. That doesn't mean the adherents are not thinking/believing, communicating, and organizing in a religious style or manner, that just means you looked at them from a non-religious perspective.

Some historical religions being a little hard to believe in the modern scientific world, doesn't mean all religions are false; quite possibly we will have self driving cars in the future. That future success won't be a disproof of past observational behavior, under the "walks like a duck, quacks like a duck, it IS a duck" doctrine of what is a religion. Praying to each other for deliverance in the form of a self driving car won't bring it here any faster, but if it does somehow arrive anyway, the physical manifestation of that idol/miracle doesn't prove the praying was the cause of the effect/idol/miracle.

Reinforcement learning isn't magic pixie dust that suddenly solves AI.

One thing we often do is assume for some compact subproblem that AI will offer massive optimizations relative to our current human baseline. However it often turns out that engineers have spent enough time to do a good enough job that the relative improvement is either not existant or small when focused on optimizing subcomponents of a system. With your video game reference there are some games that q learning does better than humans and others that does worse. For the factory floor I know data scientists and industrial engineers run optimizations to increase productivity and reduce costs. With their constraints it becomes hard or impossible for any algorithm to find a much more optimal solution especially when their problem is convex where the human generated solution is provably globally optimal.