Agree with everything you said, but I want to dig into why I don't find the critique compelling.
In short, I see a need for us all to inherently distrust anything single thing we read -- everything needs to be verified and checked out across multiple (hopefully diverse) sources of information before it can be considered "truth" in any sense. I do develop trust in certain sources over time, but meta-sources like an AI or a search engine don't get this treatment because their scope is too broad. As an example, I tend to trust folks like Fabien Sanglard[0], Eric Goldman[1], and Adam Langley[2].
Example: over on the Morrowind subreddit a few years ago, an artist had been painting landscapes of the game "Morrowind" (oil on canvas, IIRC), and when he went to print them as posters to sell to fans, the site he wanted to print them on warned him that printing his own paintings of vistas from Morrowind was copyright infringement (it's not). He came to the Morrowind subreddit for help because he'd been actively misled.
Sites like plagiarism.org[3] confidently say false things:
> But can words and ideas really be stolen?
> According to U.S. law, the answer is yes. The expression of original ideas is considered intellectual property and is protected by copyright laws, just like original inventions. Almost all forms of expression fall under copyright protection as long as they are recorded in some way (such as a book or a computer file).
This is confidently incorrect in a couple of different ways. It might seem like incompetence (and may be), but it's worth noting that plagiarism.org is actually run by TurnItIn, which profits from people not really understanding how copyright works.
Similar example (I tend to look into copyright, trademark, and patent issues a lot): I got into a discussion with another HNer about IP law, and they cited an article written by an actual lawyer[4] on a site called "The National Law Review" that's simply wrong about IP law to defend their position. It's tragic, because we've built this sort of cargo cult around "citation, please?" but then all-too-often don't keep our critical thinking skills engaged as we evaluate the source.
This is true all over. I've read HN daily for more than 10 years. Not only does every comment need scrutiny (even if stated confidently), but most stories linked on the front page do as well. It's not just blog posts or opinion pieces; some scientific studies that are posted get torn apart in the comments, to enough of a degree that, after cross-checking, I tend to think the study itself is deeply flawed.
And there are numerous stories about how Google's Quick Answers confidently give the wrong answer, as highlighted in 2017 by the WSJ[5].
I'm not saying accuracy doesn't matter, but I feel the "ChatGPT is confidently wrong" needs more context or comparative analysis around it before it becomes a compelling argument, since "confidently wrong" applies to search engines and humans as well. I haven't seen any in-depth studies on this, but would love to.
[0]: https://fabiensanglard.net/
[1]: https://blog.ericgoldman.org/
[2]: https://www.imperialviolet.org/
[3]: https://www.plagiarism.org/article/what-is-plagiarism
[4]: https://www.natlawreview.com/article/enforce-your-intellectu...
[5]: https://www.wsj.com/articles/googles-featured-answers-aim-to...