Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Being “Confidently Wrong” is holding AI back

promptql.io

Post reply on HN
Print view View original

111–120 of 274 posts

Re: Being “Confidently Wrong” is holding AI back

#111

Post by traceroute66 »

fuzzzerd wrote: post #98
gavinray wrote: post #69

Earlier quoted context omitted.

> Yeah I think our jobs are safe. I give myself 6-18 months before I think top-performing LLM's can do 80% of the day-to-day issues I'm assigned. > Why doesn’t anyone acknowledge loops like this? Thisis something you run into early-on using LLM's and learn to sidestep. This looping is a sort of "context-rot" -- the agent has the problem statement as part of it's input, and then a series of incorrect solutions. Now wh…

> This looping is a sort of "context-rot" -- the agent has the problem statement as part of it's input, and then a series of incorrect solutions. While I agree, and also use your work around, I think it stands to reason this shouldn't be a problem. The context had the original problem statement along with several examples of what not to do and yet it keeps repeating those very things instead of coming up with a diffe…

> No human would keep trying one of the solutions included in the context that are marked as not valid.

Exactly. And certainly not a genius human with the memory of an elephant and a PhD in Physics .... which is what we're constantly told LLMs are. ;-)

Re: Being “Confidently Wrong” is holding AI back

#112

Post by chpatrick »

krapp wrote: post #109
chpatrick wrote: post #106

Earlier quoted context omitted.

Do humans always answer exactly the same way to the same question? No. Also you could always pick the most likely token in an LLM as well to make it deterministic if you really wanted.

That doesn't really prove anything. I could create a Markov chain with a random seed that doesn't always answer the same question the same way, but that doesn't prove the human brain works like a Markov chain with a random seed. One thing humans tend not to do is confabulate entirely to the degree that LLMs do. When humans do so, it's considered a mental illness. Simply saying the same thing in a different way is not…

I'm not making any claim about how the human brain works. The only thing I'm saying is that humans also produce somewhat randomized output for the same question, which is pretty uncontroversial I think. That doesn't mean they're unintelligent. Same for LLMs.

Re: Being “Confidently Wrong” is holding AI back

#113

Post by CodexArcanum »

ninetyninenine wrote: post #72
sfn42 wrote: post #36

Earlier quoted context omitted.

Being confidently wrong isn't even the problem. It's a symptom of the much deeper problem that these things aren't AI at all, they're just atocomplete bots good enough to kind of seem like AI. There's no actual intelligence. That's the problem.

No. The experts in the field are past this argument. People have moved on. It is clear to everyone who builds LLMs that the AI is intelligent. The algorithm was autocomplete, but we are finding as an autocomplete bot is basically autocompleting things with humanity changing intelligent content. Your opinion is a minority now and not shared by people on the forefront of building these things. Your holding onto the ini…

LLMS don't "hallucinate" they generate a stochastic sequence of plausible tokens that, in context when read by a human, are a false statement or nonsensical.

They also dont have an internal world model. Well I don't think so, but the debate is far from settled. "Experts" like the cofounders of various AI companies (whose livelihood depends on selling these things) seem to believe that. Others do not.

https://aiguide.substack.com/p/llms-and-world-models-part-1

https://yosefk.com/blog/llms-arent-world-models.html

Re: Being “Confidently Wrong” is holding AI back

#115

Post by gavinray »

fuzzzerd wrote: post #98
gavinray wrote: post #69

Earlier quoted context omitted.

> Yeah I think our jobs are safe. I give myself 6-18 months before I think top-performing LLM's can do 80% of the day-to-day issues I'm assigned. > Why doesn’t anyone acknowledge loops like this? Thisis something you run into early-on using LLM's and learn to sidestep. This looping is a sort of "context-rot" -- the agent has the problem statement as part of it's input, and then a series of incorrect solutions. Now wh…

> This looping is a sort of "context-rot" -- the agent has the problem statement as part of it's input, and then a series of incorrect solutions. While I agree, and also use your work around, I think it stands to reason this shouldn't be a problem. The context had the original problem statement along with several examples of what not to do and yet it keeps repeating those very things instead of coming up with a diffe…

  > No human would keep trying one of the solutions included in the context that are marked as not valid.
Yeah, definitely not. Thankfully for my employment status, we're not at "human" levels QUITE yet

Re: Being “Confidently Wrong” is holding AI back

#116

Post by dinfinity »

gavinray wrote: post #69
therobots927 wrote: post #45

Earlier quoted context omitted.

Yeah I think our jobs are safe. Why doesn’t anyone acknowledge loops like this? They happen all the time and I’m only using it once a week at the most

> Yeah I think our jobs are safe. I give myself 6-18 months before I think top-performing LLM's can do 80% of the day-to-day issues I'm assigned. > Why doesn’t anyone acknowledge loops like this? Thisis something you run into early-on using LLM's and learn to sidestep. This looping is a sort of "context-rot" -- the agent has the problem statement as part of it's input, and then a series of incorrect solutions. Now wh…

I concur. Something to keep in mind is that it is often more robust to pull an LLM towards the right place than to push it away from the wrong place (or more specifically, the active parts of its latent space). Sidenote: also kind of true for humans.

That means that positively worded instructions ("do x") work better than negative ones ("don't do y"). The more concepts that you don't want it to use / consider show up in the context, the more they do still tend to pull the response towards them even with explicit negation/'avoid' instructions.

I think this is why clearing all the crap from the context save for perhaps a summarizing negative instruction does help a lot.

Re: Being “Confidently Wrong” is holding AI back

#117

Post by energy123 »

rwmj wrote: post #6

Only thing? Just off the top of my head: That the LLM doesn't learn incrementally from previous encounters. That we appear to have run out of training data. That we seem to have hit a scaling wall (reflected in the performance of GPT5). I predict we'll get a few research breakthroughs in the next few years that will make articles like this seem ridiculous.

Re online learning - If I freeze 40 yo Einstein and make it so he can't form new memories beyond 5 minutes, that's still an incredibly useful, generally intelligent thing. Doesn't seem like a problem that needs to be solved on the critical path to AGI.

Re training data - We have synthetic data, and we probably haven't hit a wall. Gpt-5 was only 3.5 months after o3. People are reading too much into the tea leaves here. We don't have visibility into the cost of Gpt-5 relative to o3. If it's 20% cheaper, that's the opposite of a wall, that's exponential like improvement. We don't have visibility into the IMO/IOI medal winning models. All I see are people curve fitting onto very limited information.

Re: Being “Confidently Wrong” is holding AI back

#118

Post by paul7986 »

nijave wrote: post #65
paul7986 wrote: post #30

And being overHyped with the doom and gloom of it's affects on society. chatGPT (5) is not there especially in replacing my field and skills: graphic, web design and web development. The first 2 there it spits out solid creations per your prompt request yet can not edit it's creations just creates new ones lol. So it's just another tool in my arsenal not a replacement to me. Such Makes me wonder how it generates the…

I don't know much about it but apparently we've been having success at work with Figma MCP hooked up to Claude in Cursor. Apparently it can pull from our component library and generate useable code (although still needs engineering to productionalize) I don't know about replacing anyone but our UI/UX designers are claiming it's significantly faster than traditional mock ups

Well until these LLMs are able to spit out initial creations it's user likes and then is able to edit it properly per each request entered into the text prompt our jobs are safe! Even better if you are also a UX Researcher along with a Designer and Developer. Research requires human interaction and AI can't touch that present to a decade or more away.

Re: Being “Confidently Wrong” is holding AI back

#119

Post by indigoabstract »

ninetyninenine wrote: post #55
lucideer wrote: post #8

While the thrust of this article is generally correct, I have two issues with it: 1. The words "the only thing" massively underplays the difficulty of this problem. It's not a small thing. 2. One of the issues I've seen with a lot of chat LLMs is their willingness to correct themselves when asked - this might seem, on the surface, to be a positive (allowing a user to steer the AI toward a more accurate or appropriate…

It’s not massively underplaying it imo. AI hype is real. This is revolutionary technology that humanity has never seen before. But it happened at a time where hype can be delivered at a magnitude never before seen by humanity as well to a degree of volume that is completely unnatural by any standard set previously by hype machines created by humanity. Not even landing on the moon has inundated people with as much hyp…

I think you would have really enjoyed living in the '50s, when the future was bright and colonizing Mars was basically a solved problem.

What we got instead is a bunch of wisecracking programmers who like to remind everyone of the 90–90 rule, or the last 10 percent.

Post reply on HN