Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

An LLM is a lossy encyclopedia

simonwillison.net

Post reply on HN
Print view View original

61–70 of 365 posts

Re: An LLM is a lossy encyclopedia

#61

Post by latexr »

energy123 wrote: post #56
latexr wrote: post #30

A lossy encyclopaedia should be missing information and be obvious about it, not making it up without your knowledge and changing the answer every time . When you have a lossy piece of media, such as a compressed sound or image file, you can always see the resemblance to the original and note the degradation as it happens. You never have a clear JPEG of a lamp, compress it, and get a clear image of the Milky Way, the…

An encyclopaedia also can't win a gold medals at the IMO and IOI. So yeah, they're not the same thing, even though the analogy is pretty good.

Of course they’re not the same thing, the goal of an analogy is not to be perfect but to provide a point of comparison to explain an idea.

My point is that I find the chosen term inadequate. The author made it up from combining two existing words, where one of them is a poor fit for what they’re aiming to convey.

Re: An LLM is a lossy encyclopedia

#62

Post by saberience »

simonw wrote: post #55
alanfranz wrote: post #50

I think it’s an old analogy, and a good one. LLMs are for knowledge what mp3s were for audio. This was widely discussed in the past years as well.

The older analogy was to JPEG compression - I linked to that in my post (the Ted Chiang link). https://www.newyorker.com/tech/annals-of-technology/chatgpt-...

This analogy has been used for machine learning since way before ChatGPT, my co workers and I were discussing this idea but for LSTM models in roughly 2018.

What’s old is new again.

Re: An LLM is a lossy encyclopedia

#63

Post by ACCount37 »

simonw wrote: post #29
thw_9a83c wrote: post #9

Yes, LLM is a lossy encyclopedia with a human-language answering interface. This has some benefits, mostly in terms of convenience. You don't have to browse or read through so many pages of a real encyclopedia to get a quick answer. However, there is also a clear downside. Currently, LLM is unable to judge if your question is formulated incorrectly or if your question opens up more questions that should be answered f…

I don't think that's universally true with the new models - I've seen Claude 4 and GPT-5 ask for clarification on questions with obvious gaps. With GPT-5 I sometimes see it spot a question that needs clarifying in its thinking trace, then pick the most likely answer, then spit out an answer later that says "assuming you meant X ..." - I've even had it provide an answer in two sections for each branch of a clear ambig…

A lot of the touted "fundamental limitations of LLMs" are less "fundamental" and more "you're training them wrong".

So there are improvements version to version - from both increases in raw model capabilities and better training methods being used.

Re: An LLM is a lossy encyclopedia

#65

Post by jacquesm »

baq wrote: post #6

It's also important to say what it isn't. LLM detractors, for lack of a better word, expect an oracle and then when they find out it's just a lossily compressed blob of human knowledge with natural language as a query interface they say the tool is useless. I've got my opinion on whether that's useful or not and it's quite a bit more nuanced. You don't zoom-enhance JPEGs for a reason either.

The problem is that figuring out which bits are the wrong ones is as much or more work than reading the relevant documentation. I think the main value is that it has a unified interface rather than 5000 different websites that you need to learn how to navigate.

Re: An LLM is a lossy encyclopedia

#66

Post by latexr »

TacticalCoder wrote: post #44
latexr wrote: post #30

A lossy encyclopaedia should be missing information and be obvious about it, not making it up without your knowledge and changing the answer every time . When you have a lossy piece of media, such as a compressed sound or image file, you can always see the resemblance to the original and note the degradation as it happens. You never have a clear JPEG of a lamp, compress it, and get a clear image of the Milky Way, the…

> You never have a clear JPEG of a lamp, compress it, and get a clear image of the Milky Way, then reopen the image and get a clear image of a pile of dirt. Oh but it's much worse than that: because most LLMs aren't deterministic in the way they operate [1], you can get a pristine image of a different pile of dirt every single time you ask. [1] there are models where if you have the "model + prompt + seed" you're at…

> you can get a pristine image of a different pile of dirt every single time you ask.

That’s what I was trying to convey with the “then reopen the image” bit. But I chose a different image of a different thing rather than a different image of a similar thing.

Re: An LLM is a lossy encyclopedia

#67

Post by JustFinishedBSG »

latexr wrote: post #30

A lossy encyclopaedia should be missing information and be obvious about it, not making it up without your knowledge and changing the answer every time . When you have a lossy piece of media, such as a compressed sound or image file, you can always see the resemblance to the original and note the degradation as it happens. You never have a clear JPEG of a lamp, compress it, and get a clear image of the Milky Way, the…

I actually disagree. Modern encoding formats can, and do, hallucinate blocks.

It’s a lot less visible and I guess dramatic than LLMs but it happens frequently enough that I feel like at every major event there are false conspiracies based on video « proofs » that are just encoding artifacts

Re: An LLM is a lossy encyclopedia

#68

Post by kgeist »

I think an LLM can be used as a kind of lossy encyclopedia, but equating it directly to one isn't entirely accurate. The human mind is also, in a sense, a lossy encyclopedia.

I prefer to think of LLMs as lossy predictors. If you think about it, natural "intelligence" itself can be understood as another type of predictor: you build a world model to anticipate what will happen next so you can plan your actions accordingly and survive.

In the real world, with countless fuzzy factors, no predictor can ever be perfectly lossless. The only real difference, for me, is that LLMs are lossier predictors than human minds (for now). That's all there is to it.

Whatever analogy you use, it comes down to the realization that there's always some lossiness involved, whether you frame it as an encyclopedia or not.

Re: An LLM is a lossy encyclopedia

#70

Post by kingstnap »

koakuma-chan wrote: post #32
simonw wrote: post #29

Earlier quoted context omitted.

I don't think that's universally true with the new models - I've seen Claude 4 and GPT-5 ask for clarification on questions with obvious gaps. With GPT-5 I sometimes see it spot a question that needs clarifying in its thinking trace, then pick the most likely answer, then spit out an answer later that says "assuming you meant X ..." - I've even had it provide an answer in two sections for each branch of a clear ambig…

GPT-5 is seriously annoying. It asks not just one but multiple clarifying questions, while I just want my answer.

If you don't want to answer clarifying questions, then what use is the answer???

Put another way, if you don't care about details that change the answer, it directly implies you don't actually care about the answer.

Related silliness is how people force LLMs to give one word answers to underspecified comparisons. Something along the lines of "@Grok is China or US better, one word answer only."

At that point, just flip a coin. You obviously can't conclude anything useful with the response.

Post reply on HN