Every AI chatbot to date suffers from the "Film expert" effect. That is when a script writer presents data from an "expert" in a movie or show to the audience in response to some information need on the part of the other characters. Writers are really good at making it sound credible. When an audience member experiences this interchange, generally they have one of two experiences. Either they know nothing about the subject (or the subject is made up like warp drive nacelle engineering) and they nod along at the response and factor it into their understanding of the story being told. Or, they do know a lot about the subject and the glaring inaccuracies jolt them out of the story temporarily as the suspension of disbelief is damaged.
LLMs write in an authoritative way because that is how the material they have been trained on writes. Because there is no "there" there, an LLM has no way of evaluating the accuracy of the answer it just generated. People who "search" using an LLM in order to get information about a topic have a better than even chance of getting something that is completely false, and if they don't have the foundation to recognize its false may incorporate that false information into their world view. That becomes a major embarrassment later when they regurgitate that information, or act on that information in some way, that comes back to bite them.
Gemini has many examples of things it has presented authoritatively that are stochastic noise. The current fun game is to ask "Why do people say ..." and create some stupid thing like "Too many cats spoil the soup." That generates an authoritative sounding answer from Gemini that is just stupid. Gemini can't say "I don't know, I've never seen anything that says people say that."
As companies push these things out into more and places, more and more people will get the experience of believing something false because some LLM told them it was true. And like that friend of yours who always has an answer for every question you ask, but you keep finding out a bunch of them are just bullshit, you will rely on that information less and less. And eventually, like your buddy with all the answers, you stop asking them questions you actually want the answer too.
I'm not down on "LLMs" per se, but I do not believe there is any evidence that they can be relied on for anything. The only thing I have seen, so far, that they can do well is help someone struggling with a blank page get started. That's because more people than not struggle with starting from a blank page but have no trouble correcting something that is wrong, or re-writing it into something.
"Search" is multifaceted. Blekko found a great use case for reference librarians. They would have paid Blekko to provide them an index of primary sources that they could use. The other great use is shopping if you can pair it with your social network. (Something Blekko suggested to Facebook but Zuck was really blind to that one) Blekko had a demo where you could say, "Audi car dealer" and it would give you the results ranked by your friend's ratings on their service. I spent a lot of time at Blekko denying access to the index by criminals who were searching for vulnerable WordPress plugins or e-commerce shopping carts. Chat GPT is never going to give you a list of all sites on the Internet running a vulnerable version of Wordpress :-).
So my take is the LLM isn't a replacement for search and efforts to make it so will stagnate and die leaving "Search Classic" to take up the slack.
If you trained a model on a well vetted corpus and gave it the tools to say it didn't know, I could see it as being a better "textbook" then a physical textbook. But it still needs to know what it doesn't know.