A "language model" describes itself already; it is a "model" of human languages.
An LLM is a lossy encyclopedia
151–160 of 365 posts
Re: An LLM is a lossy encyclopedia
#152Earlier quoted context omitted.
I don't think a "true lossy encylopedia" is a thing that has ever existed.
One could argue that’s what a pocket encyclopaedia (those exist) is. But even if we say they don’t, when you make up a term by mushing two existing words together it helps if the term makes sense . Otherwise, why even use the existing words? You called it a “lossy enyclopedia” and not a “spaghetti ice cream” for a reason, presumably so the term evokes an image or concept in the mind of the reader. If it’s bringing up…
I think you're saying that "pocket encyclopedia" is one definition of "lossy encyclopedia" that may occur to people (or that may get marketed on purpose). But that's a very poor definition of LLMs. And so the danger is that people may lock onto a wildly misleading definition. Am I getting the point?
Re: An LLM is a lossy encyclopedia
#153A "JPEG for knowledge" is how I've put it.
Re: An LLM is a lossy encyclopedia
#154Earlier quoted context omitted.
A lot of the touted "fundamental limitations of LLMs" are less "fundamental" and more "you're training them wrong". So there are improvements version to version - from both increases in raw model capabilities and better training methods being used.
I'm frustrated by the number of times I encounter people assuming that the current model behavior is inevitable. There's been hundreds of billions of dollars spent on training LLMs to do specific things. What exactly they've been trained on matters; they could have been trained to do something else. Interacting with a base model versus an instruction tuned model will quickly show you the difference between the innate…
Some of the Anthropic guys have said that the core thing holding the models back is training, and they're confident the gains will keep coming as they figure out how to onboard more and more training data. So yeah, Claude might suck at reading and writing plumbing diagrams, but they claim the barrier is simply a function of training, not any kind of architectural limitation.
Re: An LLM is a lossy encyclopedia
#155Its way more than this, today I got it to write some code, and a bit was wrong, I told it why it was wrong and it said oh ok, and toddled off and fixed it - in all the places that were wrong. This is way more than an encyclopaedia could, lossless or not.
The more I use it, the more surprised I am at its capabilities, it really is like a beginner dev, but one that doesn't learn from its mistakes (yet anyway). I find myself asking it to do more and more.
Re: An LLM is a lossy encyclopedia
#156I totally agree with the author. Sadly, I feel like that's not what the majority of LLM users tend to view LLMs. And it's definitely not what AI companies marketing. > The key thing is to develop an intuition for questions it can usefully answer vs questions that are at a level of detail where the lossiness matters the problem is that in order to develop an intuition for questions that LLMs can answer, the user will…
> the user will at least need to know something about the topic beforehand. I used ChatGPT 5 over the weekend to double check dosing guidelines for a specific medication. "Provide dosage guidelines for medication [insert here]" It spit back dosing guidelines that were an order of magnitude wrong (suggested 100mcg instead of 1mg). When I saw 100mcg, I was suspicious and said "I don't think that's right" and it quickly…
I find if I force thinking mode and then force it to search the web it’s much better.
Re: An LLM is a lossy encyclopedia
#157If the world could just internalize this simple fact - it's a search engine of all of the knowledge it has ingested, with a human-language interface (and quite a good one at that). It doesn't think. It isn't intelligence. Calling it "AI" from the beginning was a grift from the companies developing the tech.
Since late 2022, I've used LLMs extensively for coding, copywriting, research, and everything in between and I've slowly gone from "this is amazing" to "this is extremely useful but probably extremely overhyped" to "this might not actually be all that useful at all". Where accuracy matters, fact checking these things takes as much time as just doing the work manually. I think its most useful application is as a tool for spammers and bots, and that doesn't exactly bode well for the companies spending hundreds of billions of dollars on the tech.
Re: An LLM is a lossy encyclopedia
#158A lossy encyclopaedia should be missing information and be obvious about it, not making it up without your knowledge and changing the answer every time . When you have a lossy piece of media, such as a compressed sound or image file, you can always see the resemblance to the original and note the degradation as it happens. You never have a clear JPEG of a lamp, compress it, and get a clear image of the Milky Way, the…
Lossy compression does make things up. We call them compression artefacts. In compressed audio these can be things like clicks and boings and echoes and pre-echoes. In compressed images they can be ripply effects near edges, banding in smoothly varying regions, but there are also things like https://www.dkriesel.com/en/blog/2013/0802_xerox-workcentres... where one digit is replaced with a nice clean version of a diff…
> Lossy compression does make things up. We call them compression artefacts.
I don’t think this is a great analogy.
Lossy compression of images or signals tends to throw out information based on how humans perceive it, focusing on the most important perceptual parts and discarding the less important parts. For example, JPEG essentially removes high frequency components from an image because more information is present with the low frequency parts. Similarly, POTS phone encoding and mp3 both compress audio signals based on how humans perceive audio frequency.
The perceived degradation of most lossy compression is gradual with the amount of compression and not typically what someone means when they say “make things up.”
LLM hallucinations aren’t gradual and the compression doesn’t seem to follow human perception.
Re: An LLM is a lossy encyclopedia
#159Earlier quoted context omitted.
> the user will at least need to know something about the topic beforehand. I used ChatGPT 5 over the weekend to double check dosing guidelines for a specific medication. "Provide dosage guidelines for medication [insert here]" It spit back dosing guidelines that were an order of magnitude wrong (suggested 100mcg instead of 1mg). When I saw 100mcg, I was suspicious and said "I don't think that's right" and it quickly…
Using a LLM for medical research is just as dangerous as Googling it. Always ask your doctors!
This is the terrifying part: doctors do this too! I have an MD friend that told me she uses ChatGPT to retrieve dosing info. I asked her to please, please not do that.
Re: An LLM is a lossy encyclopedia
#160It’s a lossy encyclopedia that can lie to and manipulate you. In that use case, it’s fairly useless because you cannot intrinsically trust its answers without performing additional testing and research, in which case you would’ve been better off learning new things than making sure an LLM wasn’t lying to you.
> It’s a lossy encyclopedia that can lie to and manipulate you. So can a traditional encyclopedia.
We're at such a strange point where even school children knows that something like Wikipedia isn't necessarily factually correct and that you need to double check. They then go and ask ChatGPT, as if it wasn't trained on Wikipedia.
We haven't reached the stage yet where the majority of people are as sceptical of chatbots as they are of Wikipedia.
I get that even if people know not to trust a wiki, they might anyway, because, meh, good enough, but I still like us to move into a stage where the majority is at least somewhat aware that the chatbot might be wrong.