Claude is still not perfect at reading and interpreting noisy graphical data (imagine something like an EKG or chromosomal microarray plot). Still better than an average person but makes mistakes, not sure if this fits your description.
Ask HN: What is one simple thing LLMs are insanely bad at?
11–20 of 80 posts
Re: Ask HN: What is one simple thing LLMs are insanely bad at?
#12They don't generate keyword search queries very well. They can overcome this by brute force but if you watch what they search you will cringe. nhl toronto scores nhl hockey toronto scores "nhl hockey" toronto score today nhl "hockey score toronto" "hockey" who won toronto etc. Somehow being good at semantic search makes them bad at keyword search, for whatever reason.
I’ve noticed this too but it hasn’t been obvious to me that this style of search is not a learned behavior. Tool calling is very much part of the post training phase, I would expect that these style searches just naturally emerge during training. This is just my prior though.
Re: Ask HN: What is one simple thing LLMs are insanely bad at?
#13Playing Chess without letting it write a chess engine.
Re: Ask HN: What is one simple thing LLMs are insanely bad at?
#14science?!! but I'm working to fix that...
Re: Ask HN: What is one simple thing LLMs are insanely bad at?
#15It's dishonest. On several occasions team members have asked Claude to do things like analyze Gitlab CI timings and a lot of the numbers are outright fabricated. Said team members assume the numbers are good and continue with their work. Some hours are spent. Then finally someone realizes that the numbers don't look quite right and confronts Claude. Claude melts down and admits that it made it all up.
You wouldn't tolerate this kind of duplicity from a human coworker, but AI is so fast and efficient at lying, so it's OK.
Re: Ask HN: What is one simple thing LLMs are insanely bad at?
#16Video game tips. Constant mistakes and hallucinations, in my experience. Seen this across a lot of different games. Even in really well documented games, such as OSRS (which has multiple fantastic wikis).
Anno 1800 was a recent one I had trouble with, using Claude Opus. Completely made up game mechanics. Rainbow Six Siege, too.
Re: Ask HN: What is one simple thing LLMs are insanely bad at?
#17These models seem to be bad at writing prose or text. Many of the sentence structures seem to be unvaried.
If I am relying on the model to do the writing without any context or learning on how I want it to write then yes. However if I build skills that have learnt how to write in the way I want them to then I find they write very well, or at the least how I want them to as opposed to how they do natively.
Re: Ask HN: What is one simple thing LLMs are insanely bad at?
#18Short answers to simple questions.
Accurate short answers / text are always harder than long answers, for human or AI. I know several authors and editors who write a lot longer at first, then spend a multiple of the initial time compressing it via a back and forth process to something dense. Sort of like weaving the initial threads.
I found this can work with AI. You get it to generate a lot more at first, and then do several passes over it to compress and squeeze out the noise while keeping the core information. With AI, at least with my prompts, it takes some effort (on my end) to get it to really really cut down the noise and not cut everything out.
Re: Ask HN: What is one simple thing LLMs are insanely bad at?
#19providing value for the actual cost (not the price we're being charged atm, the actual cost)
Re: Ask HN: What is one simple thing LLMs are insanely bad at?
#20humor
+1
We need humor benchmarks!