31–40 of 174 posts
You just have to double check the results whenever you tell Claude to extract lists and data. 99.9% of it will be correct, but sometimes 1 or 2 records are off. This kind of error is especially hard to notice because you're so impressed that Claude managed to do the extract task at all -- plus the results look wholly plausible upon eyeballing -- that you wouldn't expect anything to be wrong at all. But LLMs can get t…
You just have to double check the results whenever you tell Claude to extract lists and data. 99.9% of it will be correct, but sometimes 1 or 2 records are off. This kind of error is especially hard to notice because you're so impressed that Claude managed to do the extract task at all -- plus the results look wholly plausible upon eyeballing -- that you wouldn't expect anything to be wrong at all. But LLMs can get t…
Example, I was trying out two podcast apps and wanted to get a diff of the feeds I had subscribed to. I initially asked the LLM to compare the two OPML files but it got the results wrong. I could have spent the next 30 minutes prompt engineering and manually verifying results, but instead I asked it to write a script to compare two LLMs, which turned out fine. It's fairly easy to inspect a script and be confident it's _probably_ accurate compared to the tedious process of checking a complex output.
You just have to double check the results whenever you tell Claude to extract lists and data. 99.9% of it will be correct, but sometimes 1 or 2 records are off. This kind of error is especially hard to notice because you're so impressed that Claude managed to do the extract task at all -- plus the results look wholly plausible upon eyeballing -- that you wouldn't expect anything to be wrong at all. But LLMs can get t…
You just have to double check the results whenever you tell Claude to extract lists and data. 99.9% of it will be correct, but sometimes 1 or 2 records are off. This kind of error is especially hard to notice because you're so impressed that Claude managed to do the extract task at all -- plus the results look wholly plausible upon eyeballing -- that you wouldn't expect anything to be wrong at all. But LLMs can get t…
They are just human.
Also they've been trained to say something as plausible as possible. If it happens to be true, then that's great because it's extra plausible. If it's not true, no big deal.
While I have worked with one awful human in the past who was like that, most thankfully aren't!
I'm finding Claude to hallucinate less than ChatGPT, and to be far more accurate at coding than CoPilot. Pleasantly surprised on both counts. Example hallucinations from ChatGPT include researching the dates of historical events for the company I work at, trivially verifiable by me but I was being lazy... ChatGPT told me about blog posts that never existed and I could prove never existed, Claude was spot on with date…
( He did not mention anything about ChatGPT, but him using Claude instead says a lot )
My takeaway: Time for me to move from ChatGPT to Claude
Earlier, a product manager sent me a list of company id's to turn on a feature flag for as a screenshot. Rather than enter them all in manually, I used that to get GPT4-o to generate a comma separated list from the screenshot. Again, worked perfectly.
I've been using ChatGPT for similar visual recognition things. Recently I took a video of a car because I really liked its color. I upladed the video to ChatGPT and asked to extract which paint color I'd need to specify to a modding garage to put a foil on my car. ChatGPT really impressed me, extracted a screenshot, did a color analysis of the paint, found the palettes from paint vendors and found for me the exact pa…
I had a similar requirement a few days back, but I stopped at Google lens. TIL
I did a similar thing with ChatGPT-4o giving it a txt file list of movie names, and it returned the release date of each one, then gave me an iCal file of all those movies with their release dates as recurring yearly anniversary events so I can watch my favorite movies on their anniversary. I’ve done it for a few friends as well now and it’s got a 100% success rate so far, across over 100 total movie names.
To bad that wouldn't work for good music albums. For many years (still, maybe?) albums would come out on Tuesdays. So one day of the week you'd have them all bunched up together. That's how I remember that September 11, 2001 was a Tuesday. Album day, and there was a good one that day, too.
You just have to double check the results whenever you tell Claude to extract lists and data. 99.9% of it will be correct, but sometimes 1 or 2 records are off. This kind of error is especially hard to notice because you're so impressed that Claude managed to do the extract task at all -- plus the results look wholly plausible upon eyeballing -- that you wouldn't expect anything to be wrong at all. But LLMs can get t…
Isn't asking an LLM "to write a Python validator" suffers from the 99.9% (or whatever the error rate for validators written by Claude) problem?
Also, the LLM is more likely to fail spectacularly by hallucinating APIs when writing a script, and more likely to fail subtly on parsing tasks.