Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Being “Confidently Wrong” is holding AI back

promptql.io

Post reply on HN
Print view View original

141–150 of 274 posts

Re: Being “Confidently Wrong” is holding AI back

#141

Post by vidarh »

therobots927 wrote: post #45
traceroute66 wrote: post #32

Earlier quoted context omitted.

> is their willingness to correct themselves when asked Except they don't correct themselves when asked. I'm sure we've all been there, many, many, many,many,many times .... - User: "This is wrong because X" - AI: "You're absolutely right ! Here's a production-ready fixed answer" - User: "No, that's wrong because Y" - AI: "I apologise for frustrating you ! Here's a robust answer that works" - User: "You idiot, you ju…

Yeah I think our jobs are safe. Why doesn’t anyone acknowledge loops like this? They happen all the time and I’m only using it once a week at the most

Because it's easy to learn to stop engaging with those loops, treating them as a sign you provided too little context, and instead start a new conversation with an expanded prompt.

It doesn't mean these loops aren't an issue, because they are, but once you stop engaging with them and cut them off, they're a nuisance rather than a showstopper.

Re: Being “Confidently Wrong” is holding AI back

#142

Post by delichon »

gavinray wrote: post #133
yifanl wrote: post #132

Earlier quoted context omitted.

There are people convinced that if we throw a sufficient amount of training data and VC money at more hardware, we'll overcome the gap. Technically, I can't prove that they're wrong, novel solutions sometimes happen, and I guess the calculus is that it's likely enough to justify a trillion dollars down the hole.

There's a guy, Ken Stanley, who wrote the NEAT[0]/HyperNEAT[1] algorithms. His big idea is that evolution/advancements don't happen incrementally, but rather in unpredictable large leaps. He wrote a whole book about it that's pretty solid IMO: "Why Greatness Cannot Be Planned: The Myth of the Objective." [0] https://en.wikipedia.org/wiki/Neuroevolution_of_augmenting_t... [1] https://en.wikipedia.org/wiki/HyperNEAT

https://en.wikipedia.org/wiki/Saltation_(biology)

Re: Being “Confidently Wrong” is holding AI back

#143

Post by vidarh »

fuzzzerd wrote: post #98
gavinray wrote: post #69

Earlier quoted context omitted.

> Yeah I think our jobs are safe. I give myself 6-18 months before I think top-performing LLM's can do 80% of the day-to-day issues I'm assigned. > Why doesn’t anyone acknowledge loops like this? Thisis something you run into early-on using LLM's and learn to sidestep. This looping is a sort of "context-rot" -- the agent has the problem statement as part of it's input, and then a series of incorrect solutions. Now wh…

> This looping is a sort of "context-rot" -- the agent has the problem statement as part of it's input, and then a series of incorrect solutions. While I agree, and also use your work around, I think it stands to reason this shouldn't be a problem. The context had the original problem statement along with several examples of what not to do and yet it keeps repeating those very things instead of coming up with a diffe…

I agree it shouldn't be a problem, but if you don't regularly run into humans who insist on trying solutions clearly signposted as wrong or not valid, you're far luckier than I am.

Re: Being “Confidently Wrong” is holding AI back

#144

Post by nyeah »

stetrain wrote: post #126
nyeah wrote: post #77

Earlier quoted context omitted.

Yes, that is the point of the comment.

Yes, you’re absolutely right! Agreeing with the comment and adding my own experience was the point of my comment. Is there anything else I can help you with?

Ok, fair, clearly I misinterpreted what you wrote.

Re: Being “Confidently Wrong” is holding AI back

#145

Post by IsTom »

projektfu wrote: post #29
roxolotl wrote: post #11

The big thing here is that they can’t even be confident. There is no there there. They are a, admittedly very useful, statistical model. Ascribing confidence to it is an anthropomorphizing mistake which is easy to make since we’re wired to trust text that feels human. They are at their most useful when it is cheaper to verify their output than it is to generate it yourself. That’s why code is rather ok; you can run i…

The article buries the lede by waiting until the very end to talk about solutions like having the LLM write DSL code. Presumably if you feed an LLM your orders table and a question about it, you'll get an answer that you can't trust. But if you ask it to write some SQL or similar thing based on your database to get the answer and run it, you can have more confidence.

Until it mishandles a NULL somewhere in a condition on does JOIN instead of a LEFT JOIN and outputs something plausibly-looking that is just plain wrong. To verify it you'll need to do the work that it would take to write it anyway.

Re: Being “Confidently Wrong” is holding AI back

#146

Post by psadauskas »

vidarh wrote: post #139
ACCount37 wrote: post #68

Earlier quoted context omitted.

1-turn instruction following and multi-turn instruction following are not the same exact capability, and some AIs only "get good" at the former. 1-turn gets more training attention - because it's more noticeable, in casual use and benchmarks both, and also easier to train for. With weak multi-turn instruction following, context data will often dominate over user instructions. Resulting in very "loopy" AI - and more s…

This is a good point, and to drive this home to people, if you have a conversation of this pattern: User: Fix this problem ... Assistant: X User: No, don't do X Assistant: Y User: No, Y is wrong too. Assistant: X It is generally pointless to continue. You now have a context that is full of the assistant explaining to you and itself why X and Y are the right answers, and much less context of you explaining why it is w…

There's also the Pink Elephant Paradox (Whatever you do, DO NOT think about a pink elephant).

If you mention X or Y, even if they're preceded by "DO NOT" in all caps, an LLM will still end up with both X and Y into its context, making it more likely it gets used.

I'm running out of ways to tell the assistant to not use mocks for tests, it really really wants to use them.

Re: Being “Confidently Wrong” is holding AI back

#147

Post by gus_massa »

corytheboyd wrote: post #127

Isn’t it obvious that the confidently wrong problem will never go away because all of this is effectively built on a statistical next token matcher? Yeah sure you can throw on hacks like RAG, more context window, but it’s still built on the same foundation. It’s like saying you built a 3D scene on a 2D plane. You can employ clever tricks to make 2D look 3D at the right angle, buts it’s fundamentally not 3D, which obv…

It's easy to solve if they modify they training to remove some weight from Stack Overflow and add more weight to Yahoo! Answers :) .

I remember a few years ago, we were planing to make some kind of math forum for students in the first year of the university. My opinion was that it was too easy to do it wrong. On one way you can be like Math Overflow were all the questions are too technical (for first year of the university) and all the answers are too technical (first year of the university). On the other way, you can be like Yahoo! Answers, where more than half of the answers were "I don't know", with many "I don't know" per question.

For the AI, you want to give it some room to generalize/bullshit. It one page says that "X was a few months before Z" and another page says that "Y was a few days before Z", than you want an hallucinated reply that says that "X happened before Y".

On the other hand, you want the AI to say "I don't know.". They just gave too little weight to the questions that are still open. Do you know a good forum where people post questions that are still open?

Re: Being “Confidently Wrong” is holding AI back

#150

Post by staticman2 »

chpatrick wrote: post #131
staticman2 wrote: post #123

Earlier quoted context omitted.

I really wish people into LLMs would limit themselves to terms from neuroscience or philosophy when descrbing humans. You are in my mind rightfully getting pushback for writing "human experts also output tokens with some statistical distribution. "

That's just a mathematical fact. You have a big opaque box with a slot where you can put text in and you can see text come out. The text that comes out follows some statistical distribution (obviously), and isn't always the same. Can you decide just from that if there's an LLM or a human sitting inside the box? No. So you can't make conclusions about whether the box as a system is intelligent just because it outputs…

Okay... I objected to your use of the word token. Humans don't think in tokens or even write in tokens so obviously what you wrote is not a fact.

That shouldn't even be controversial, I don't think?

You wrote "The text that comes out follows some statistical distribution".

At the risk of being over my head here did you mean the text can be described statistically or "follows some statistical distribution". Are these two concepts the same thing? I don't think so.

A program by design follows some statistical distribution. A human is doing whatever electrochemical thing it's doing that can be described statistically after the fact.

Regardless my point was pretty simple, I know this will never happen but I wish tech people would drop this tech language when describing humans and adopt neuroscience language.

Post reply on HN