Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Being “Confidently Wrong” is holding AI back

promptql.io

Post reply on HN
Print view View original

61–70 of 274 posts

Re: Being “Confidently Wrong” is holding AI back

#61

Post by jqpabc123 »

Being able to recall all the data from the internet doesn't make you "intelligent".

It makes you a walking database --- an example of savant syndrome.

Combine this with failure on simple logical and cognitive tests and the diagnosis would be --- idiot savant.

This is the best available diagnosis of an LLM. It excels at recall and text generation but fails in many (if not most) other cognitive areas.

But that's ok, let's use it to replace our human workers and see what happens. Only an idiot would expect this to go well.

https://nypost.com/2024/06/17/business/mcdonalds-to-end-ai-d...

Re: Being “Confidently Wrong” is holding AI back

#63

Post by Wowfunhappy »

traceroute66 wrote: post #32
lucideer wrote: post #8

While the thrust of this article is generally correct, I have two issues with it: 1. The words "the only thing" massively underplays the difficulty of this problem. It's not a small thing. 2. One of the issues I've seen with a lot of chat LLMs is their willingness to correct themselves when asked - this might seem, on the surface, to be a positive (allowing a user to steer the AI toward a more accurate or appropriate…

> is their willingness to correct themselves when asked Except they don't correct themselves when asked. I'm sure we've all been there, many, many, many,many,many times .... - User: "This is wrong because X" - AI: "You're absolutely right ! Here's a production-ready fixed answer" - User: "No, that's wrong because Y" - AI: "I apologise for frustrating you ! Here's a robust answer that works" - User: "You idiot, you ju…

...I don't know why, but I swear to god, when Claude gets into one of these cycles I can often get it out by dropping the f-bomb, with maybe a 50% success rate. Something about that word lets it know that it needs to break the pattern.

Re: Being “Confidently Wrong” is holding AI back

#64

Post by JCM9 »

Add to being confidently wrong is the super annoying way it corrects itself after disastrously screwing something up.

AI: “I’ve deployed the API data into your app, following best practices and efficient code.”

Me: “Nope thats totally wrong and in fact you just wrote the API credential into my code, in plaintext, into the JavaScript which basically guarantees that we’re gonna get hacked.”

AI: “You’re absolutely right. Putting API credentials into the source code for the page is not a best practice, let me fix that for you.”

Re: Being “Confidently Wrong” is holding AI back

#65

Post by nijave »

paul7986 wrote: post #30

And being overHyped with the doom and gloom of it's affects on society. chatGPT (5) is not there especially in replacing my field and skills: graphic, web design and web development. The first 2 there it spits out solid creations per your prompt request yet can not edit it's creations just creates new ones lol. So it's just another tool in my arsenal not a replacement to me. Such Makes me wonder how it generates the…

I don't know much about it but apparently we've been having success at work with Figma MCP hooked up to Claude in Cursor. Apparently it can pull from our component library and generate useable code (although still needs engineering to productionalize)

I don't know about replacing anyone but our UI/UX designers are claiming it's significantly faster than traditional mock ups

Re: Being “Confidently Wrong” is holding AI back

#66

Post by CloseChoice »

LLMs are largely used by developers, who (in some sense or the other) supervise what the LLM does constantly (even if that means for sum committing to main and running in production). We do already have a lot of tools: tests, compilation, a programming language with its harsh restrictions compared to natural language, and of course the eye test, this is not the case for a lot of jobs where GenAI is used for hyperautomation, so I am really curious in which way it will or won't get adopted in other areas.

Re: Being “Confidently Wrong” is holding AI back

#67

Post by Zigurd »

Zigurd wrote: post #35
chpatrick wrote: post #26

Earlier quoted context omitted.

Sure, it's only as good as the training data. But human experts also output tokens with some statistical distribution. That doesn't mean anything.

That sounds plausible. But it doesn't explain why LLM's make laughably bad errors that even a biased and haphazard human researcher wouldn't make.

Gemini seems to have a user interface that, for the way most people encounter Gemini, is more closely linked to search results. This leads me to suspect that Google's approach to training could be uniquely informed by both current and historic web crawling.

Re: Being “Confidently Wrong” is holding AI back

#68

Post by ACCount37 »

traceroute66 wrote: post #32
lucideer wrote: post #8

While the thrust of this article is generally correct, I have two issues with it: 1. The words "the only thing" massively underplays the difficulty of this problem. It's not a small thing. 2. One of the issues I've seen with a lot of chat LLMs is their willingness to correct themselves when asked - this might seem, on the surface, to be a positive (allowing a user to steer the AI toward a more accurate or appropriate…

> is their willingness to correct themselves when asked Except they don't correct themselves when asked. I'm sure we've all been there, many, many, many,many,many times .... - User: "This is wrong because X" - AI: "You're absolutely right ! Here's a production-ready fixed answer" - User: "No, that's wrong because Y" - AI: "I apologise for frustrating you ! Here's a robust answer that works" - User: "You idiot, you ju…

1-turn instruction following and multi-turn instruction following are not the same exact capability, and some AIs only "get good" at the former. 1-turn gets more training attention - because it's more noticeable, in casual use and benchmarks both, and also easier to train for.

With weak multi-turn instruction following, context data will often dominate over user instructions. Resulting in very "loopy" AI - and more sessions that are easier to restart from scratch than to "fix".

Gemini is notorious for underperforming at this, while Claude has relatively good performance. I expect that many models from lesser known providers would also have a multi-turn instruction following gap.

Re: Being “Confidently Wrong” is holding AI back

#69

Post by gavinray »

therobots927 wrote: post #45
traceroute66 wrote: post #32

Earlier quoted context omitted.

> is their willingness to correct themselves when asked Except they don't correct themselves when asked. I'm sure we've all been there, many, many, many,many,many times .... - User: "This is wrong because X" - AI: "You're absolutely right ! Here's a production-ready fixed answer" - User: "No, that's wrong because Y" - AI: "I apologise for frustrating you ! Here's a robust answer that works" - User: "You idiot, you ju…

Yeah I think our jobs are safe. Why doesn’t anyone acknowledge loops like this? They happen all the time and I’m only using it once a week at the most

  > Yeah I think our jobs are safe.
I give myself 6-18 months before I think top-performing LLM's can do 80% of the day-to-day issues I'm assigned.

  > Why doesn’t anyone acknowledge loops like this?
Thisis something you run into early-on using LLM's and learn to sidestep. This looping is a sort of "context-rot" -- the agent has the problem statement as part of it's input, and then a series of incorrect solutions.

Now what you've got is a junk-soup where the original problem is buried somewhere in the pile.

Best approach I've found is to start a fresh conversation with the original problem statement and any improvements/negative reinforcements you've gotten out of the LLM tacked on.

I typically have ChatGPT 5 Thinking, Claude 4.1 Opus, Grok 4, and Gemini 2.5 Pro all churning on the same question at once and then copy-pasting relevant improvements across each.

Re: Being “Confidently Wrong” is holding AI back

#70

Post by nijave »

Zigurd wrote: post #9
chpatrick wrote: post #4

Earlier quoted context omitted.

It is if the weights are sufficiently advanced.

The weights, so to speak, come from the knowledge base. That means you can't get away from the quality of the knowledge base. That isn't uniform across all domains of knowledge. Then the problem becomes how do you make the training material uniformly high-quality in every knowledge domain? At best it becomes the meta problem of determining the quality of knowledge in some way that makes an LLM able to calibrate confi…

MCP and agents seem like a solutions but as far as I know maintaining sufficient context is still a problem

I.e. ability to plug in expert data sources

Post reply on HN