Does anyone know which XKCD comic the top image was? Or was it just created in the style of XKCD?
Being “Confidently Wrong” is holding AI back
171–180 of 274 posts
Re: Being “Confidently Wrong” is holding AI back
#172Wow, there really is an xkcd for everything.
Re: Being “Confidently Wrong” is holding AI back
#173While the thrust of this article is generally correct, I have two issues with it: 1. The words "the only thing" massively underplays the difficulty of this problem. It's not a small thing. 2. One of the issues I've seen with a lot of chat LLMs is their willingness to correct themselves when asked - this might seem, on the surface, to be a positive (allowing a user to steer the AI toward a more accurate or appropriate…
Being confidently wrong isn't even the problem. It's a symptom of the much deeper problem that these things aren't AI at all, they're just atocomplete bots good enough to kind of seem like AI. There's no actual intelligence. That's the problem.
My favorite "paper" on AI pretty accurately describes this line of thinking
Re: Being “Confidently Wrong” is holding AI back
#174What's really funny to me is, sometimes it fixes itself if you just ask "are you SURE ABOUT THIS ANSWER?" myself and others often wonder, why the heck don't they run a 2nd model to "proofread" output or spot check it. Like did you actually answer the question or are you going off a really weird tangent. I asked Perplexity some question for sample UI code for Rust / Slint, it gave me a beautiful web UI, I think it got…
Because that would be twice as computationally intensive.
"Reasoning" models integrate some of that natively. In a way, they're trained to double check themselves - which does improve accuracy at the cost of compute.
Re: Being “Confidently Wrong” is holding AI back
#175The big thing here is that they can’t even be confident. There is no there there. They are a, admittedly very useful, statistical model. Ascribing confidence to it is an anthropomorphizing mistake which is easy to make since we’re wired to trust text that feels human. They are at their most useful when it is cheaper to verify their output than it is to generate it yourself. That’s why code is rather ok; you can run i…
But as a statistical model, it should be able to report some notion of statistical uncertainty, not necessarily in its next-token outputs, but just as a separate measure. Unfortunately, there really doesn't seem to be a lot of effort going into this.
The statistical certainty is indeed present in the model. Each token comes with a probablility; if your softmax results approach a uniform distribution (i.e. all selected tokens at the given temp have near equal probabilities), then the next most likely token is very uncertain. Reporting the probabilities of the returned tokens can help the user understand how likely hallucinations are. However, that information is deliberately obfuscated now, to prevent distillation techniques.
Re: Being “Confidently Wrong” is holding AI back
#176Isn’t it obvious that the confidently wrong problem will never go away because all of this is effectively built on a statistical next token matcher? Yeah sure you can throw on hacks like RAG, more context window, but it’s still built on the same foundation. It’s like saying you built a 3D scene on a 2D plane. You can employ clever tricks to make 2D look 3D at the right angle, buts it’s fundamentally not 3D, which obv…
I mean if it's trained on things like Reddit then it's just reflecting its training data. I asked a question on reddit just yesterday and the only response I got was confidently wrong. This is not the first time it has happened.
Re: Being “Confidently Wrong” is holding AI back
#177Earlier quoted context omitted.
Okay... I objected to your use of the word token. Humans don't think in tokens or even write in tokens so obviously what you wrote is not a fact. That shouldn't even be controversial, I don't think? You wrote "The text that comes out follows some statistical distribution". At the risk of being over my head here did you mean the text can be described statistically or "follows some statistical distribution". Are these…
> Humans don't think in tokens or even write in tokens so obviously what you wrote is not a fact. Doesn't matter what they think in. A token can be a letter or a word or a sound. The point is that the box takes some sequence of tokens and produces some sequence of tokens. > You wrote "The text that comes out follows some statistical distribution". > At the risk of being over my head here did you mean the text can be…
I think we are talking past each other but this has been entertaining.
I'd say anybody who writes "the LLM just produces characters with some stochastic process, therefore it's not intelligent or correct" is making an implicit argument about the way the LLM works and the way the human brain works. There might even be an implicit argument about how intelligence works.
They are not making the argument that you can't make up statistical models to describe a box, a human generated text, or an expert human opinion. But that seems to be the claim you are responding to.
Re: Being “Confidently Wrong” is holding AI back
#178Earlier quoted context omitted.
1-turn instruction following and multi-turn instruction following are not the same exact capability, and some AIs only "get good" at the former. 1-turn gets more training attention - because it's more noticeable, in casual use and benchmarks both, and also easier to train for. With weak multi-turn instruction following, context data will often dominate over user instructions. Resulting in very "loopy" AI - and more s…
This is a good point, and to drive this home to people, if you have a conversation of this pattern: User: Fix this problem ... Assistant: X User: No, don't do X Assistant: Y User: No, Y is wrong too. Assistant: X It is generally pointless to continue. You now have a context that is full of the assistant explaining to you and itself why X and Y are the right answers, and much less context of you explaining why it is w…
> I see so many people get stuck "arguing" with a model over this, getting more and more frustrated as the model keeps repeating variations of the broken answer
Maybe because people expect AI systems that are touted as all-knowing, all-powerful, coming-for-your-job to be smart enough to remember what was said two turns ago?
Re: Being “Confidently Wrong” is holding AI back
#179Well fucking yeah
Yesterday I asked ChatGPT a really simple, factual question. "Where is this feature on this software?" And it made up a menu that didn't exist. I told "No,, you're hallucinating, search the internet for the correct answer" and it directly responded (without the time delay and introspection bubbles that indicate an internet search) "That is not a hallucination, that is factually correct". God damn.
Re: Being “Confidently Wrong” is holding AI back
#180Earlier quoted context omitted.
No. The experts in the field are past this argument. People have moved on. It is clear to everyone who builds LLMs that the AI is intelligent. The algorithm was autocomplete, but we are finding as an autocomplete bot is basically autocompleting things with humanity changing intelligent content. Your opinion is a minority now and not shared by people on the forefront of building these things. Your holding onto the ini…
> Like you realize humans hallucinate too right? A developer that hallucinates at work to the extent that LLMs does would probably have issues getting their PRs past code reviews a lot.
They would have issues even remaining employed. AI defenders are very quick to point out "humans mistakes too", but that is a false equivalence because humans learn. If a junior makes a really stupid mistake, when I show him the correct way he won't make that mistake again. An AI will, because (as people correctly point out) it has no actual intelligence.