Basic rule of MLE is to have guardrails on your model output; you don't want some high-leverage training data point to trigger problems in prob. These guardrails should be deterministic and separate from the inference system, and basically a stack of user-defined policies. LLMs are ultimately just interpolated surfaces and the rules are the same as if it were LOESS.
The "confident idiot" problem: Why AI needs hard rules, not vibe checks
221–230 of 399 posts
Re: The "confident idiot" problem: Why AI needs hard rules, not vibe checks
#222Earlier quoted context omitted.
> Except in cases where the training data is more wrong than correct (e.g. niche expertise where the vox pop is wrong) Same for human knowledge though. Learn from society/school/etc that X is Y, and you repeat X is Y, even if it's not. > However, an LLM no more deals in Q&A than in facts. It only typically replies to a question with an answer because that itself is statistically most likely, and the words of the answ…
We likely learn/generate language in an auto-regressive way at least conceptually similar to an LLM, but this isn't just self-contained auto-regressive generation... Humans use language to express something (facts, thoughts, etc), so you can consider these thoughts being expressed as a bias to the language generation process, similar perhaps to an image being used as a bias to the captioning part of an image captioni…
>Humans use language to express something (facts, thoughts, etc), so you can consider these thoughts being expressed as a bias to the language generation process
My point however is more that the "thoughts being expressed" are themselves being generated by a similar process (and that it's either that or a God-given soul).
Re: The "confident idiot" problem: Why AI needs hard rules, not vibe checks
#223Ironic considering how many LLMs are competing to be trained on Reddit . . . which is the biggest repository of confidently incorrect people on the entire Internet. And I'm not even talking politics. I've lost count of how much stuff I've seen there related to things I can credibly professionally or personally speak to that is absolute, unadulterated misinformation and bullshit. And this is now LLM training data.
One thing I've had to explain to many confused friends who use reddit is that many of the people presenting themselves as domain experts in subreddits related to fields like law, accounting, plumbing, electrical, construction, etc. have absolutely no connection to or experience in whatever the field is.
I had a co-worker talk once about how awesome Reddit was and how much life advice she'd taken from it and I was just like . . . yeah . . .
Re: The "confident idiot" problem: Why AI needs hard rules, not vibe checks
#224- Claude, please optimise the project for performance. o Claude goes away for 15 minutes, doesn't profile anything, many code changes. o Announces project now performs much better, saving 70% CPU. - Claude, test the performance. o Performance is 1% _slower_ than previous. - Claude, can I have a refund for the $15 you just wasted? o [Claude waffles], "no".
You need to let it actually benchmark. They are only as good as the tools you give them.
Re: The "confident idiot" problem: Why AI needs hard rules, not vibe checks
#225we're inching towards the three laws of robotics
Re: The "confident idiot" problem: Why AI needs hard rules, not vibe checks
#226The thing that bothers me the most about LLMs is how they never seem to understand "the flow" of an actual conversation between humans. When I ask a person something, I expect them to give me a short reply which includes another question/asks for details/clarification. A conversation is thus an ongoing "dance" where the questioner and answerer gradually arrive to the same shared meaning. LLMs don't do this. Instead,…
Have you used Claude much? It often responds to things with questions
Re: The "confident idiot" problem: Why AI needs hard rules, not vibe checks
#227Earlier quoted context omitted.
> but simply because they operate at the level of words, not facts. They are language models. Facts can be encoded as words. That's something we also do a lot for facts we learn, gather, and convey to other people. 99% of university is learning facts and theories and concept from reading and listening to words. Also, even when directly observing the same fact, it can be interpreted by different people in different wa…
Animal/human brains and LLMs have fundamentally different goals (or loss functions, if you prefer), even though both are based around prediction. LLMs are trained to auto-regressively predict text continuations. They are not concerned with the external world and any objective experimentally verifiable facts - they are just self-predicting "this is what I'm going to say next", having learnt that from the training data…
>Humans/animals are embodied, living in the real world, whose design has been honed by a "loss function" favoring survival. Animals are "designed" to learn facts about the real world, and react to those facts in a way that helps them survive.
Yes - but LLMs also get this "embodied knowledge" passed down from human-generated training data. We are their sensory inputs in a way (which includes their training images, audio, and video too).
They do learn in a batch manner, and we learn many things not from books but from a more interactive direct being in the world. But after we distill our direct experiences and throughts derived from them as text, we pass them down to the LLMs.
Hey, there's even some kind of "loss function" in the LLM case - from the thumbs up/down feedback we are asked to give to their answers in Chat UIs, to $5/hour "mechanical turks" in Africa or something tasked with scoring their output, to rounds of optimization and pruning during training.
>Animals are predicting something EXTERNAL (facts) vs LLMs predicting something INTERNAL (what will I say next).
I don't think that matters much, in both cases it's information in, information out.
Human animals predict "what they will say/do next" all the time, just like they also predict what they will encounter next ("my house is round that corner", "that car is going to make a turn").
Our prompt to an LLM serves the same role as sensory input from the external world plays to our predictions.
Re: The "confident idiot" problem: Why AI needs hard rules, not vibe checks
#228Confident idiot (an LLM) writes an article bemoaning confident idiots.
[flagged]
Not 100% sure I understand your comment, but just to make sure my stance is clear - I saw that it was AI-written and noped out. Thought it was a little funny that they used an LLM to write an article about how LLMs are bad.
Re: The "confident idiot" problem: Why AI needs hard rules, not vibe checks
#229Earlier quoted context omitted.
Yes you're totally right! I misunderstood what you meant, let me write six more paragraphs based on a similar misunderstanding rather than just trying to get clarification from you
My favorite is when it bounces back and forth between the same two wrong answers, each time admitting that the most recent answer is wrong and going back to the previous wrong answer. Doesn't matter if you tell it "that's not correct and neither is ____ so don't try that instead," it likes those two answers and it's going to keep using them.
The false info baked into its context at that point in the conversation and it will get stuck in a local minima trying to generate a response to the given context.
Re: The "confident idiot" problem: Why AI needs hard rules, not vibe checks
#230Earlier quoted context omitted.
This is a recent phenomenon. It seems most of the pages today are SEO optimized LLM garbage with the aim of having you scroll past three pages of ads. THe internet really used to be efficient and i could always find exactly what i wanted with an imprecise google search ~ 15 years ago.
You'd think with the reputation of LLMs being trained on Twitter (pre-Musk radicalization) and Reddit, they'd be better at understanding normal conversation flow since twitter requires short responses and Reddit... while Wall of Text happens occasionally, it's not the typical cadence of the discussion.
Twitter, Reddit, HN don't always have the consistency of conversation that two people talking do.
Even here, I'm responding to you on a thread that I haven't been in on previously.
There's also a lot more material out there in the format of Stack Exchange questions and answers, Quora posts, blog posts and such than there is for consistent back and forth interplay between two people.
IRC chat logs might have been better...ish.
The cadence for discussion is unique to the medium in which the discussion happens. What's more, the prompt may require further investigation and elaboration prior to a more complete response, while other times it may be something that requires story telling and making it up as it goes.