Being “Confidently Wrong” is holding AI back

Post by therobots927 » Fri, Aug 22, 2025, 1:14 PM UTC

RyanOD wrote: post #74

therobots927 wrote: post #45

Earlier quoted context omitted.

Yeah I think our jobs are safe. Why doesn’t anyone acknowledge loops like this? They happen all the time and I’m only using it once a week at the most

But still under pressure in the short-term, no? As companies lean into AI as a means of efficiency / competitive advantage / cost savings, jobs will be eliminated / reduced while companies find their direction. The potential gains are said to be too big to sit on the sidelines and wait to be a late-adopter.

Post by lucideer » Fri, Aug 22, 2025, 1:18 PM UTC

traceroute66 wrote: post #32

lucideer wrote: post #8

While the thrust of this article is generally correct, I have two issues with it: 1. The words "the only thing" massively underplays the difficulty of this problem. It's not a small thing. 2. One of the issues I've seen with a lot of chat LLMs is their willingness to correct themselves when asked - this might seem, on the surface, to be a positive (allowing a user to steer the AI toward a more accurate or appropriate…

> is their willingness to correct themselves when asked Except they don't correct themselves when asked. I'm sure we've all been there, many, many, many,many,many times .... - User: "This is wrong because X" - AI: "You're absolutely right ! Here's a production-ready fixed answer" - User: "No, that's wrong because Y" - AI: "I apologise for frustrating you ! Here's a robust answer that works" - User: "You idiot, you ju…

Post by jqpabc123 » Fri, Aug 22, 2025, 1:19 PM UTC

JCM9 wrote: post #64

Add to being confidently wrong is the super annoying way it corrects itself after disastrously screwing something up. AI: “I’ve deployed the API data into your app, following best practices and efficient code.” Me: “Nope thats totally wrong and in fact you just wrote the API credential into my code, in plaintext, into the JavaScript which basically guarantees that we’re gonna get hacked.” AI: “You’re absolutely right…

Post by Zigurd » Fri, Aug 22, 2025, 1:23 PM UTC

nijave wrote: post #70

Zigurd wrote: post #9

Earlier quoted context omitted.

The weights, so to speak, come from the knowledge base. That means you can't get away from the quality of the knowledge base. That isn't uniform across all domains of knowledge. Then the problem becomes how do you make the training material uniformly high-quality in every knowledge domain? At best it becomes the meta problem of determining the quality of knowledge in some way that makes an LLM able to calibrate confi…

MCP and agents seem like a solutions but as far as I know maintaining sufficient context is still a problem I.e. ability to plug in expert data sources

Post by taco_emoji » Fri, Aug 22, 2025, 1:24 PM UTC

decentrality wrote: post #40

lucideer wrote: post #8

While the thrust of this article is generally correct, I have two issues with it: 1. The words "the only thing" massively underplays the difficulty of this problem. It's not a small thing. 2. One of the issues I've seen with a lot of chat LLMs is their willingness to correct themselves when asked - this might seem, on the surface, to be a positive (allowing a user to steer the AI toward a more accurate or appropriate…

Agreed with #1 ( came here to say that also ) Pronoun and noun wordplay aside ( 'Their' ... `themselves` ) I also agree that LLMs can correct the path being taken, regenerate better, etc... But the idea that 'AI' needs to be _stubbornly_ wrong ( more human in the worst way ) is a bad idea. There is a fundamental showing, and it is being missed. What is the context reality? Where is this prompt/response taking place?…