We need a Butlerian Jihad against AI

I'm looking forward to it, but I reserve the right to cling to the 1984 release.

Regulating research into general AI would not make those barriers considerably higher.

Tesla critics and competitors are also strongly in favor of self-driving regulations.

> Far more important than the process: strong AI is immoral in and of itself. For example, if you have strong AI, what are you going to do with it besides effectively have robotic slaves? And even if, by some miracle, you create strong AI in a mostly ethical way, and you also deploy it in a mostly ethical way, strong AI is immoral just in its existence. I mean that it is an abomination. It’s not an evolved being.

My fear is that most software, even when useful, locks us into certain paths. Our situations or needs change, evolve, but we will remain subject to inflexible software, to systems we cannot make change with us, in the vast majority of cases. Only a very few programs strive for better: spreadsheets being one noted example.

Ursala Franklin categorized technology as holistic or prescriptive[1], where it is something wielded or something that directs us. Even a social media app which lets us create content- a seemingly holistic act- still has narrow prescriptive channels we can not escape. We will never be able to understand or enhance this tool. We will never understand it, never see it's nature. This, to me, is the definition of what Erik talks about: an abomination, a thing beyond comprehension, a horror outside of reality, the form of existence which is shared.

I feel like we're reaching a crisis where we are creating an unknowable, unexplorable world. We're building an anti-Enlightenment prison. That, to me, constitutes a deontological hazard, demands that we assess the action themselves of creating unexplorable software.

[Edit: I misread the line I quotes as, "what are you going to do with it besides effectively be robotic slaves": that uhh changes the pertinence of our two discussions here notably. I think it's risky that the strong ai would be used to try to architect policies/systems that steer people, which is a different concern than Erik's.]

[1] https://en.wikipedia.org/wiki/Ursula_Franklin#Holistic_and_p...

"The philosopher John Searle made precisely this argument about the standard conception of rationality. His point was that there are no odds that would rationally allow a parent to bet the life of their child for a quarter. Human nature just doesn’t work that way, and it shouldn’t work that way."

I agree that it sounds morally repugnant to risk your child's life for a quarter, but in practice people do do this all the time.

Imagine your child wants ice cream. There is some utility in taking your child to the nearby ice cream parlor. Your child will be made happy and you will be made happy by making your child happy. However, this is not infinite utility. In other words, there is probably some amount of money I could offer you to not take your child to get ice cream today. If I offered you 10,000 dollars to not get your child ice cream today, I bet the vast majority of people would take the deal. That sets the upper bound of the utility of taking your child to the ice cream parlor at 10,000 dollars.

Suppose the ice cream parlor is 3 miles away (I just checked the distance to my favorite ice cream parlor and it is 3 miles away). In the United States this website[1] says there is approximately 1 death per 100 million vehicle miles traveled. We could rephrase that as 1 * 10^-8 chance to die per vehicle mile traveled. This risk may be high, presumably you aren't drunk or impaired, maybe you're a better driver than average or have a safer or better maintained car, or live in a safer place, but the risk of death isn't zero.

If you are willing to drive your child 3 miles to go get ice cream then it seems like you are willing to expose your child to the risk of death from car accidents for utility that is less than 10,000 dollars. Putting those ideas together we could calculate the odds where a parent would, in practice, risk the life of their child for a quarter.

I don't quite know what to make of this. I tend to think that people would regard doing odds calculations like this for real life decisions as somewhat sociopathic and would just prefer to live as if significantly unlikely bad things were impossible or just refuse to think about the moral implications of probabilities. That seems similar to what the article is saying, people just prefer to live as if bad super collider or AI experiments won't happen rather than reason about them.

I tend to think that being too "reasonable" on a local scale is bad. That is, I will still take my children to go get ice cream even though I know driving is a risk. At higher levels though I want people to be making decisions that are increasingly based on reason and probabilities. I do want the traffic engineers to be reasoning about vehicle deaths per mile and the like when they are setting speed limits, traffic signs, and the like. For things like AI and, I suppose, super colliders, our decision makers should absolutely be considering things rationally.

1 - https://www.iihs.org/topics/fatality-statistics/detail/state...

Another point of nerdity that no-one has mentioned yet, including the OP: Herbert sketched out an extended story that portrays humanity and the machines it had fought against so long merging in the long run. In part this is why Leto II never destroyed Ix even though it was constantly (quietly) breaking the Bulterian Jihad rules.

None of this invalidates the OP's core point, of course. I think it's a good and valuable discussion to consider technology from fundamentally moral grounds, and I wish we'd do it more.