IMO the real danger of good AI is that we don't have the collective intelligence to safely manage our expectations writ large. This train of thought is a little messy, so apologies in advance:
- Just like the fusiform gyrus responds to human faces at a subconscious level (see: uncanny valley faces just "feeling" wrong, because they are detected as inhuman below the level of conscious thought), the Wernicke's area responds to human speech when reading text. I believe that grammatically-perfect speech is subconsciously perceived as human by most people, even tech people, despite attempts to remain impartial - we are biologically hard-wired to assign humanness to written language.
- ChatGPT and its peers do not report confidence levels, so the typical interaction is information that may or may not be correct, presented confidently.
- Average (non-tech) people interacting with chat AIs are aware that it is connected to vast stores of training data without understanding the limitations of statistical LLMs or the need for confidence values in responses, lending them an air of "intelligence" due to the volume of data "available."
- This leads to a non-trivial number of people interacting with chat AI and taking its responses as gospel.
Anecdotally, if you traverse social media you will see an unending stream of people reporting their experiences with how amazing chatGPT is, using it for everything from writing emails to writing school essays. The problem is that when a non-tech person interacts with ChatGPT, they assume based on the above listed factors that what they get back is correct, valid thought from a semi-intelligent being. Even knowing it's a robot, the perfect correctness of the grammar will instill a feeling of validity in a non-trivial segment of the population over time.
This is leading to a scenario where people trust what GPT says about various topics without bothering to question it, and I believe this is already happening. When I bring this up to other tech people, it is usually dismissed with "well, everyone knows it's just an AI," or "people will learn it's limitations." However, at the risk of being glib, consider George Carlin: "think about how dumb the average person is, and then realize half the population is dumber than that." What happens when the average person turns to a statistical LLM for advice on relationships, career moves, how to vote, or other nebulous topics where there is no real correct answer? How will we control where ChatGPT is steering vast numbers of uninformed petitioners?
We already struggle as a society with collective action on existentially important topics like controlling microplastic dispersion, regulating toxic additives in consumer products, and climate change. And those topics are "merely" complex; imagine how much harder it will be to control unintended or unforeseen consequences of a human-like intelligence-adjacent being delivering information of questionable value to an unquestioning audience of 20-40% of humanity?
Addendum: I am also very worried about schoolchildren using AI to write their essays and book reports, skipping critical reading-comprehension time and arriving at adulthood unable to comprehend anything more complex than a menu without asking AI to summarize it.