I feel like this is a feature which improves the perceived confidence of the LLM but doesn't do much for correctness of other outputs, i.e. an exacerbation of the "confidently incorrect" criticism.
A recent LinkedIn post that I came across as an example of people trusting (or learning to trust) AI too much while not realizing that it can make up numbers too: https://www.linkedin.com/posts/mariamartin1728_claude-wrote-... P.S. Credit to the poster, she posted a correction note when someone caught the issue: https://www.linkedin.com/posts/mariamartin1728_correction-on...
Honestly, people make them up just as much or generate equally incorrect graphs.
It's about time our trust into random visualizations is destroyed, without the actual formulas and data behind being exposed.