LLMs are still surprisingly bad at some simple tasks

Once again an example of "anti-ai people are those who treat LLMs as oracles, not the pro-ai people."

The point from the end of the post that AI produces output that sounds correct is exactly what I try to emphasize to friends and family when explaining appropriate uses of LLMs. AI is great at tasks where sounding correct is the essence of the task (for example "change the style of this text"). Not so great when details matter and sounding correct isn't enough, which is what the author here seems to have rediscovered.

The most effective analogy I have found is comparing LLMs to theater and film actors. Everyone understands that, and the analogy offers actual predictive power. I elaborated on the idea if you're curious to read more:

https://jstrieb.github.io/posts/llm-thespians/

I think Gemini is one of the best example of an LLM that is in some cases the best and in some cases truly the worst.

I once asked it to read a postcard written by my late grandfather in Polish, as I was struggling to decipher it. It incorrectly identified the text as Romanian and kept insisting on that, even after I corrected it: "I understand you are insistent that the language is Polish. However, I have carefully analyzed the text again, and the linguistic evidence confirms it is Romanian. Because the vocabulary and alphabet are not Polish, I cannot read it as such." Eventually, after I continued to insist that it was indeed Polish, it got offended and told me it would not try again, accusing me of attempting to mislead it.

> "Something that describes how an AI is convincing if you don't understand its reasoning, and close to useless if you understand its limitations."

This made me laugh. Because it's the exact opposite sentiment of anti-LLM crowd. So which is it? Is it only useful if you know what you're doing or less useful if you know what you're doing?

> "I can't wait until I can jack into the Metaverse and buy an NFT with cryptocurrency just by using an LLM! Perhaps I can view it on my 3D TV by streaming it over WIMAX? I'd better stock up on quantum computers to make sure it all works."

In the author's attempt to be a smartass, they showed themselves. It makes them sound childish. Instead of just admitting they were wrong, they make some flippant remark about cryptocurrency and NFT'S, despite having vastly different purposes and goals and successes. Just take the L.

to add: "I shouldn't have to know anything about LLMs to use them correctly" is one heck of a take, but ok.

> "I don't. I hate the way this is being sold as a universal and magical tool. The reality doesn't live up to the hype."

And I hate the way in which people will do the opposite: claim it has no uses cases. It's literally the same sentiment, but in reverse. It's just as myopic and naive. But for whatever reason, we can look at a CEO hawking it and think "They're just trying to make more money" but can't see the flipside of devs not wanting to lose their livelihoods to something. We have just as much to lose as they have to gain, but want to pretend like we're objective.

I wanted to check the prime factors of 1966 the other day so I googled it and it led me to https://brightchamps.com/en-us/math/numbers/factors-of-1966 , a site that seems focussed on number facts. It confidently states that prime factors of 1966 are 2, 3, 11, and 17. For fun I tried to multiply these numbers back in my head and concluded there's no way that 6 * 187 could reach 1966.

That's when I realized this site was making heavy use of AI. Sadly, lots of people are going to trust but not verify...