131–140 of 156 posts
What I find really strange about this is I use AI a lot as a “smart friend” to work through explanations of things I find difficult etc and I am currently preparing for some exams so I will often give the AI a document and ask for some supporting resources to take the subject further and it almost always produces something that is plausibly close to a real thing but wrong in specifics. As in when you ask for a refere…
What I find really strange about this is I use AI a lot as a “smart friend” to work through explanations of things I find difficult etc and I am currently preparing for some exams so I will often give the AI a document and ask for some supporting resources to take the subject further and it almost always produces something that is plausibly close to a real thing but wrong in specifics. As in when you ask for a refere…
What this story tells us more than anything is that Lindell cannot convince a competent lawyer to defend him, so what he gets instead are clownshod phonies. Either he’s out of cash, or he’s such a terrible client that nobody with a shred of professional responsibility will take him.
My social studies teacher in 8th grade throughout the year would give us a list of things and phrases by decade. Loved these assignments. Totally threw myself into them. Years later, I was working at a prep school and wanted the students to be assigned Billy Joel's We Didn't Start the Fire as a similar assignment (competition?). Teachers thought it was stupid, that the kids wouldn't like it (um, who cares?), and that it was too hard. Teacher's responses only confirmed what I thought about the school (meh) and that it was a sad day that curiosity was a bad thing. (I was the fundraiser for the school so I didn't really interact with the kids a lot but ones I knew would have had fun with such a project).
Anyway, the Lindell lawyers must have gone to this school, or one like it. How is it ever okay to do this and think it's a good idea? And, how the heck did these people pass the Bar?
Edit: List of references in We Didn't Start the Fire https://en.wikipedia.org/wiki/List_of_references_in_We_Didn%.... Gonna blog the references and this post on my little policy blog this week :-)
What I find really strange about this is I use AI a lot as a “smart friend” to work through explanations of things I find difficult etc and I am currently preparing for some exams so I will often give the AI a document and ask for some supporting resources to take the subject further and it almost always produces something that is plausibly close to a real thing but wrong in specifics. As in when you ask for a refere…
It has likely never occurred to them that such checks are necessary. Why would it, if they've never performed such checks, nor happen to have been warned by AI critics?
>For example, yesterday I got a list of some study resources for abstract algebra. Claude referred me to a series by Benedict Gross (Which is excellent btw). It gave me a line to harvard’s website but it was a 404 and it was only with further searching that I found the real thing. It also suggested a youtube playlist by Socratica (again this exists but the url was wrong) and one by Michael Penn (same deal).
FWIW, I've found Penn's content to be quite long-winded and poorly edited. The key idea being presented often makes up hardly any of the video's runtime, so I'm just sitting there watching the guy actually write out the steps solving an equation (and making trivial errors, and not always correcting them).
Earlier quoted context omitted.
What this story tells us more than anything is that Lindell cannot convince a competent lawyer to defend him, so what he gets instead are clownshod phonies. Either he’s out of cash, or he’s such a terrible client that nobody with a shred of professional responsibility will take him.
This. And only this. My social studies teacher in 8th grade throughout the year would give us a list of things and phrases by decade. Loved these assignments. Totally threw myself into them. Years later, I was working at a prep school and wanted the students to be assigned Billy Joel's We Didn't Start the Fire as a similar assignment (competition?). Teachers thought it was stupid, that the kids wouldn't like it (um,…
What I find really strange about this is I use AI a lot as a “smart friend” to work through explanations of things I find difficult etc and I am currently preparing for some exams so I will often give the AI a document and ask for some supporting resources to take the subject further and it almost always produces something that is plausibly close to a real thing but wrong in specifics. As in when you ask for a refere…
Because its makers don't care about precision or correctness. They care only about convincing the people that matter that gaping software bugs are "hallucinations" that can never be fixed 100%, and that that is an acceptable outcome.
For every story about AI-assisted legal briefs with tons of mistakes, there will be 100 PR-ridden pieces about how $LATEST_VERSION has passed the bar exam or has discovered a miracle drug. There will never be a story about AI successfully arguing a real case however, because the main goal was only about selling a vision of labor replacement. Whether or not the replacement can do the job as specified is immaterial.
Earlier quoted context omitted.
LLMs were originally designed for translation, so it makes sense. We have basically elimated the need to learn foreign languages for day to day use anyway, its only helpful for high professional tasks or close literary study or prestige.
That’s such a ridiculous statement.
> [learning a language is] only helpful for high professional tasks or close literary study or prestige.
This is a person who doesn't understand actual face-to-face communication, like, at all. Even though translation apps are amazing, in a social interaction, there's no getting over the imposition of the halting, hesitant back-and-forth of device-assisted translation. Sure, you can almost always eventually get your point across, but you're never going to set the other party at ease in the same way as speaking their language yourself.
Earlier quoted context omitted.
That's not correct, and seems to be based on a common misunderstanding of how LLMs work, the rough idea being that when the info the model is being asked for had been in the data used for training, it "looks it up" not unlike software looking up info from a huge database of general knowledge, and that when that lookup fails it falls back to making stuff up. But that's wrong, the models are actually doing the exact sa…
If an LLM happens to know the answer to your question, that answer will have the greatest weight, and will therefore become a non-hallucinated output. Otherwise the output will be hallucinated. Note that a hallucination may manifest as an attempt to extrapolate, which may be successful. If you query an LLM with prior knowledge that the LLM doesn’t know the answer, you are guaranteed to receive a hallucinated output.…
> "If an LLM happens to know the answer to your question, that answer will have the greatest weight"
An LLM doesn’t “know” anything in the way you’re imagining. It doesn’t have stored facts or indexed knowledge to check against, it just has weights learned between token sequences, and it outputs whatever next token is assigned the highest probability given the prompt and prior context. That might happen to produce a correct answer (and people are obviously working hard to make the models produce right answers as often as possible), but it might just as easily produce a plausible-sounding but wrong one, even if the correct information was in the training data. Because that correct information being there doesn't guarantee it will have the highest weighting ever, yet alone the highest weighting in all contexts of previous tokens and in all temperature settings.
You’re right that hallucinations can sometimes look like “extrapolations” that happen to land correctly, but that’s incidental. It’s still doing the same token-by-token probability selection regardless of whether it ends up right or wrong.
Framing it around “missing knowledge” vs “existing knowledge” is misleading intuition. It’s better to think about it in terms of probability distributions over token sequences: the model’s training biases it toward correct sequences more often than incorrect ones, but there’s nothing fundamental in the architecture that guarantees that if the answer was present in training, it will always beat out wrong guesses.
p.s. It's late at night here and I'm about to go to bed, so apologies if I've not explained well in this comment - I gave it to ChatGPT hoping it could tidy things up for me and it just made a way more confusing version so I'm posting it as is :D Let me know if my explanation still isn't clear and I could try again, or answer any questions you have, tomorrow
Earlier quoted context omitted.
That's not correct, and seems to be based on a common misunderstanding of how LLMs work, the rough idea being that when the info the model is being asked for had been in the data used for training, it "looks it up" not unlike software looking up info from a huge database of general knowledge, and that when that lookup fails it falls back to making stuff up. But that's wrong, the models are actually doing the exact sa…
If an LLM happens to know the answer to your question, that answer will have the greatest weight, and will therefore become a non-hallucinated output. Otherwise the output will be hallucinated. Note that a hallucination may manifest as an attempt to extrapolate, which may be successful. If you query an LLM with prior knowledge that the LLM doesn’t know the answer, you are guaranteed to receive a hallucinated output.…
You're missing the point. It doesn't "know" anything. The only thing it can "know" is the statistical relationships between tokens in its dataset. It doesn't "know" anything about the meaning of those tokens. It doesn't even "know" whether it "knows" anything or not. The best it can do is "Here's a recursively generated string of ASCII codes that are statistically likely to follow each other according to the data corpus."
It's Rashomon. It can point you in the right directions a lot of the time, but there's no getting around the fact that you have to double-check its answers with external sources.
> Or at least this is how I interpret the term.
That's not a very useful interpretation because it's not grounded in technical reality.
Earlier quoted context omitted.
If an LLM happens to know the answer to your question, that answer will have the greatest weight, and will therefore become a non-hallucinated output. Otherwise the output will be hallucinated. Note that a hallucination may manifest as an attempt to extrapolate, which may be successful. If you query an LLM with prior knowledge that the LLM doesn’t know the answer, you are guaranteed to receive a hallucinated output.…
> If an LLM happens to know the answer to your question You're missing the point. It doesn't "know" anything. The only thing it can "know" is the statistical relationships between tokens in its dataset. It doesn't "know" anything about the meaning of those tokens. It doesn't even "know" whether it "knows" anything or not. The best it can do is "Here's a recursively generated string of ASCII codes that are statistical…
The word know is an abstraction I use in order to avoid going into technical details.
> That's not a very useful interpretation because it's not grounded in technical reality.
My interpretation aligns with what people generally mean by hallucination, and it's definitely more useful than saying that any output is hallucination.
Earlier quoted context omitted.
People are lazy. I’m enrolled in a language class in a foreign country right now - so presumably people taking that class want to actually get good at the language so they can actually live their life here - yet a significant portion of students just turn in ChatGPT essays. And I don’t mean essays edited with chatGPT, but essays that are clearly verbatim output. When the teacher asks the students to read them out lou…
LLMs were originally designed for translation, so it makes sense. We have basically elimated the need to learn foreign languages for day to day use anyway, its only helpful for high professional tasks or close literary study or prestige.