Earlier quoted context omitted.
I understand your sentiment, but I also don't think it's fair to say people are being disingenuous. I don't work for an AI company, I just use it with decent results. My last project needed a way to tag certain types of business activity indicated in 17,000 user reviews. I wrote a prompt with 5 different tags along with descriptions for each business activity, took a few-shot approach by defining 8 different examples…
(Reader beware: basically everything the LLM emits in here is wrong.) I was attempting to find formats that used a particular quoting style — SQL style double-the-character-to-espace — and bizarrely JSON came up. Gemini: > JSON: In JSON, double quotes are used to delimit strings. If a double quote needs to be included within the string itself, it's escaped by doubling it (e.g., "This is a string with a "quote" in it.…
I had a situation where I was playing around with having it right a quick method for me. I gave it the specs it needed to work with and the other methods it will call.
Basically, it wrote calling one of the methods with the wrong (missing) arguments. When I asked it to fix that call, it first said its current call was right but then when it went to explain it (in the same message), it pointed out that a certain missing argument was required to be passed in.
I have seen similar situations with these models looking at data time and time again. It theoretically sees it, but once that wrong data (the first failed interpretation) is part of its context it's basically all over.