Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Being “Confidently Wrong” is holding AI back

promptql.io

Post reply on HN
Print view View original

131–140 of 274 posts

Re: Being “Confidently Wrong” is holding AI back

#131

Post by chpatrick »

staticman2 wrote: post #123
chpatrick wrote: post #112

Earlier quoted context omitted.

I'm not making any claim about how the human brain works. The only thing I'm saying is that humans also produce somewhat randomized output for the same question, which is pretty uncontroversial I think. That doesn't mean they're unintelligent. Same for LLMs.

I really wish people into LLMs would limit themselves to terms from neuroscience or philosophy when descrbing humans. You are in my mind rightfully getting pushback for writing "human experts also output tokens with some statistical distribution. "

That's just a mathematical fact.

You have a big opaque box with a slot where you can put text in and you can see text come out. The text that comes out follows some statistical distribution (obviously), and isn't always the same.

Can you decide just from that if there's an LLM or a human sitting inside the box? No. So you can't make conclusions about whether the box as a system is intelligent just because it outputs characters in a stochastic manner according to some distribution.

Re: Being “Confidently Wrong” is holding AI back

#132

Post by yifanl »

corytheboyd wrote: post #127

Isn’t it obvious that the confidently wrong problem will never go away because all of this is effectively built on a statistical next token matcher? Yeah sure you can throw on hacks like RAG, more context window, but it’s still built on the same foundation. It’s like saying you built a 3D scene on a 2D plane. You can employ clever tricks to make 2D look 3D at the right angle, buts it’s fundamentally not 3D, which obv…

There are people convinced that if we throw a sufficient amount of training data and VC money at more hardware, we'll overcome the gap.

Technically, I can't prove that they're wrong, novel solutions sometimes happen, and I guess the calculus is that it's likely enough to justify a trillion dollars down the hole.

Re: Being “Confidently Wrong” is holding AI back

#133

Post by gavinray »

yifanl wrote: post #132
corytheboyd wrote: post #127

Isn’t it obvious that the confidently wrong problem will never go away because all of this is effectively built on a statistical next token matcher? Yeah sure you can throw on hacks like RAG, more context window, but it’s still built on the same foundation. It’s like saying you built a 3D scene on a 2D plane. You can employ clever tricks to make 2D look 3D at the right angle, buts it’s fundamentally not 3D, which obv…

There are people convinced that if we throw a sufficient amount of training data and VC money at more hardware, we'll overcome the gap. Technically, I can't prove that they're wrong, novel solutions sometimes happen, and I guess the calculus is that it's likely enough to justify a trillion dollars down the hole.

There's a guy, Ken Stanley, who wrote the NEAT[0]/HyperNEAT[1] algorithms.

His big idea is that evolution/advancements don't happen incrementally, but rather in unpredictable large leaps.

He wrote a whole book about it that's pretty solid IMO: "Why Greatness Cannot Be Planned: The Myth of the Objective."

[0] https://en.wikipedia.org/wiki/Neuroevolution_of_augmenting_t... [1] https://en.wikipedia.org/wiki/HyperNEAT

Re: Being “Confidently Wrong” is holding AI back

#134

Post by ForHackernews »

gavinray wrote: post #69
therobots927 wrote: post #45

Earlier quoted context omitted.

Yeah I think our jobs are safe. Why doesn’t anyone acknowledge loops like this? They happen all the time and I’m only using it once a week at the most

> Yeah I think our jobs are safe. I give myself 6-18 months before I think top-performing LLM's can do 80% of the day-to-day issues I'm assigned. > Why doesn’t anyone acknowledge loops like this? Thisis something you run into early-on using LLM's and learn to sidestep. This looping is a sort of "context-rot" -- the agent has the problem statement as part of it's input, and then a series of incorrect solutions. Now wh…

> I give myself 6-18 months before I think top-performing LLM's can do 80% of the day-to-day issues I'm assigned.

How long before there's an AI smart enough to say 'no' to half the terrible ideas I'm assigned?

Re: Being “Confidently Wrong” is holding AI back

#135

Post by aleph_minus_one »

jansper39 wrote: post #97
therobots927 wrote: post #45

Earlier quoted context omitted.

Yeah I think our jobs are safe. Why doesn’t anyone acknowledge loops like this? They happen all the time and I’m only using it once a week at the most

Honestly when I speak about these sorts of issues I get the feeling that other people view me as some kind of luddite, especially people above me who presumably want to replace as many people with AI as possible. I suppose me pointing out the flaws breaks the illusion of magic that people want AI to have.

> I suppose me pointing out the flaws breaks the illusion of magic that people want AI to have.

My impression is rather: there exist two kinds of people who are "very invested in this illusion":

1. People who want to get rich by either investing in or working on AI-adjacent topics. They of course have an interest to uphold this illusion of magic.

2. People who have a leftist agenda ("we will soon all be replaced by AI, so politics has to implement [leftist policy measures like UBI]"). If people realize that AI is not so powerful, after all, such leftist political measures whose urgency was argued with the (hypothetical) huge societal changes that will be caused by AI will not have a lot backing in society, or at least not considered to be urgently implemented by society.

Re: Being “Confidently Wrong” is holding AI back

#136

Post by rtkwe »

ACCount37 wrote: post #68
traceroute66 wrote: post #32

Earlier quoted context omitted.

> is their willingness to correct themselves when asked Except they don't correct themselves when asked. I'm sure we've all been there, many, many, many,many,many times .... - User: "This is wrong because X" - AI: "You're absolutely right ! Here's a production-ready fixed answer" - User: "No, that's wrong because Y" - AI: "I apologise for frustrating you ! Here's a robust answer that works" - User: "You idiot, you ju…

1-turn instruction following and multi-turn instruction following are not the same exact capability, and some AIs only "get good" at the former. 1-turn gets more training attention - because it's more noticeable, in casual use and benchmarks both, and also easier to train for. With weak multi-turn instruction following, context data will often dominate over user instructions. Resulting in very "loopy" AI - and more s…

I have this problem all the time with minor image edits on ChatGPT the few time's I've tried it. Any time I try to do a second edit or change to the generated image it seems to take the already degraded output from it's first attempt and use that instead of the original image.

Re: Being “Confidently Wrong” is holding AI back

#138

Post by tango12 »

lucideer wrote: post #8

While the thrust of this article is generally correct, I have two issues with it: 1. The words "the only thing" massively underplays the difficulty of this problem. It's not a small thing. 2. One of the issues I've seen with a lot of chat LLMs is their willingness to correct themselves when asked - this might seem, on the surface, to be a positive (allowing a user to steer the AI toward a more accurate or appropriate…

The AI being wrong problem is probably not insurmountable.

Humans have meta-cognition that helps them judge if they're doing a thing with lots of assumptions vs doing something that's blessed.

Humans decouple planning from execution right? Not fully but we choose when to separate it and when to not.

If we had enough data on here's a good plan given user context and here's a bad plan, it doesn't seem unreasonable to have a pretty reliable meta cognition capability on the goodness of a plan.

Re: Being “Confidently Wrong” is holding AI back

#139

Post by vidarh »

ACCount37 wrote: post #68
traceroute66 wrote: post #32

Earlier quoted context omitted.

> is their willingness to correct themselves when asked Except they don't correct themselves when asked. I'm sure we've all been there, many, many, many,many,many times .... - User: "This is wrong because X" - AI: "You're absolutely right ! Here's a production-ready fixed answer" - User: "No, that's wrong because Y" - AI: "I apologise for frustrating you ! Here's a robust answer that works" - User: "You idiot, you ju…

1-turn instruction following and multi-turn instruction following are not the same exact capability, and some AIs only "get good" at the former. 1-turn gets more training attention - because it's more noticeable, in casual use and benchmarks both, and also easier to train for. With weak multi-turn instruction following, context data will often dominate over user instructions. Resulting in very "loopy" AI - and more s…

This is a good point, and to drive this home to people, if you have a conversation of this pattern:

    User: Fix this problem ...
    Assistant: X
    User: No, don't do X
    Assistant: Y
    User: No, Y is wrong too.
    Assistant: X
It is generally pointless to continue. You now have a context that is full of the assistant explaining to you and itself why X and Y are the right answers, and much less context of you explaining why it is wrong.

If you reach that state, start over, and constrain your initial request to exclude X and Y. If it brings up either again, start over, and constrain your request further.

If the model is bad at handling multiple turns without getting into a loop, telling it that it is wrong is not generally going to achieve anything, but starting over with better instructions often will.

I see so many people get stuck "arguing" with a model over this, getting more and more frustrated as the model keeps repeating variations of the broken answer, without realising they're filling the context with arguments from the model for why the broken answer is right.

Re: Being “Confidently Wrong” is holding AI back

#140

Post by yifanl »

gavinray wrote: post #133
yifanl wrote: post #132

Earlier quoted context omitted.

There are people convinced that if we throw a sufficient amount of training data and VC money at more hardware, we'll overcome the gap. Technically, I can't prove that they're wrong, novel solutions sometimes happen, and I guess the calculus is that it's likely enough to justify a trillion dollars down the hole.

There's a guy, Ken Stanley, who wrote the NEAT[0]/HyperNEAT[1] algorithms. His big idea is that evolution/advancements don't happen incrementally, but rather in unpredictable large leaps. He wrote a whole book about it that's pretty solid IMO: "Why Greatness Cannot Be Planned: The Myth of the Objective." [0] https://en.wikipedia.org/wiki/Neuroevolution_of_augmenting_t... [1] https://en.wikipedia.org/wiki/HyperNEAT

I would suspect that any next step comes with a novel implementation though, not just trying to scale the same shit to infinity.

I guess the bitter lesson is gospel now, which doesn't sit right with me now that we're past the stage of Moore's Law being relevant, but I'm not the one with a trillion dollars, so I don't matter.

Post reply on HN