>
The danger of computers becoming like humans is not as great as the danger of humans becoming like computers.-- Konrad Zuse
What eats me is trajectory we are on as humans. Runaway actual intelligence, even if it destroys humanity, wouldn't worry me as much, I'd wish it good luck, IMO even a totally random dice roll is better than what we're aiming at. But AI is more a meme than even an honest intent. It's like saying "I really really want blueberry pie", but then when you ask what that is, they it gets real murky real quick, but that doesn't stop the hype, as if wanting something a lot makes up for not knowing what it is. But that doesn't prime a person to make blueberry pie, it primese them to get lured by what they think is the smell of blueberry pie.
Here's something to note, as the length of a discussion about "AI" grows in length, the probability of things getting explained via something they saw in a movie or read in a book or saw on TV, glossing over 99.9% of the "details" those left out, approaches 1. You may say we make this fiction because of our achievements, or may point to things that actually did come to pass (of course, compared to the stuff that didn't, even from the same authors, it's nothing). And I love using examples, too, and I sure love quotes.
But still, I think when we are this steeped in variations of the same thing over and over and over, of course we'll "consider it" at some point, and the moral or philoshopical depth is drastically reduced by already being primed. We're like people who don't see what we build with our hands, because we wear VR googles that show us movies of our childhood or some console game.
What I can see us realistically making are are "idols" with eyes that do not see, with audio output, perfect speech synthesis, that does not convey meaning, incredibly fast analysis that is not thought. From the get go, starting with the Turing test, it was more about how what something seems from the outside, than what it is to itself on the inside.
Furthermore, we might make human level AI no problem, EZ PZ, but not by making AI so smart, but my making humans dumber. We're already training ourselves to select what we consume and think from discrete pre-configured options. We notice and complain about the effects of in all sorts of smaller areas, but it's a general trend, and I think it's not so much about creating something "better" than humans, but about removing human agency.
> The frightening coincidence of the modern population explosion with the discovery of technical devices that, through automation, will make large sections of the population 'superfluous' even in terms of labor, and that, through nuclear energy, make it possible to deal with this twofold threat by the use of instruments beside which Hitler's gassing installations look like an evil child's fumbling toys, should be enough to make us tremble.
-- Hannah Arendt
Meanwhile, there's this idea that humans becoming "superfluous" means we'll all be free from "bad" work, and free for fun work and leisure. How we would get from an increasing concentration of wealth in fewer hands to some commnuist utopia? Is that some kind of integer overflow, where enough greed and power wrap over to sharing and letting others live and decide their own fate? We're connected to that (like Michael Scott is to the baby of his boss,) by delusion, the path we're on doesn't lead there.
Throw away a word here, do something that "everybody does" there, adapt to "how the world is" some, and there you go, a blank nothing that can be deprecated without guilt or resistance. The desire to control human agency is met more than halfway by our desire to shed it, to abdicate responsibility, become a piece of floatsam flowing down the river of history to the ocean of technotopia, enter the holy land of holodeck, where we can consume endlessly. We digitize, we sample, that's how we make things "manageable", and at high enough resolution we can fool ourselves, or have something "good enough to work with".
And just like children that get too much sugar too early tend to not liking fruit as much, because they're not as extremely sweet, our abstractions lure some people to prefer them over the dirty, fractal, infinite real world, or the exchange of emojis and pre-configured figures of speech over real human contact, silence that isn't awkward, thinking about what you're trying to say, or even coming up blank and that being okay... just like we go "posterized, high contrast" in all sorts of ways aready, I hve no problem supposing that we will come up with a form of alienation like that, but for thinking, I just no clue how it will look like.
We already have it with language of course, but I'm sure we can take that to the next level, maybe neural interfaces. If we can't read and transmit thoughts in their fullness and depth, then hey, just reduce our thoughts to the equivalent of grunts, that might work. Become like a computer, 0 and 1. Convince yourself that's that just what humans have been all along, remember Star Trek wisdom, don't be so proud and consider your brain more than a "meat machine", don't deny Data his quest to become human! Cue super emotional music swelling up.