An LLM is a lossy encyclopedia • Hacker News Board

Re: An LLM is a lossy encyclopedia

#21

Post by baq » Tue, Sep 02, 2025, 10:09 AM UTC

littlestymaar wrote: post #14

baq wrote: post #6

It's also important to say what it isn't. LLM detractors, for lack of a better word, expect an oracle and then when they find out it's just a lossily compressed blob of human knowledge with natural language as a query interface they say the tool is useless. I've got my opinion on whether that's useful or not and it's quite a bit more nuanced. You don't zoom-enhance JPEGs for a reason either.

> You don't zoom-enhance JPEGs for a reason either. Tell that to the Google Pixel product team: https://mk.absturztau.be/notes/ac4jnvjpskjc02lq

That's both perfect for this thread and terrible in general.

If you can see the analogy between text and pictures, it drives the point exactly the right way: in both cases you expect a database to know things it either can't or forgot. If it had a good picture of the zoomed in background it could probably generate a very good representation of what the cropped part would look like; same thing works with text.

Re: An LLM is a lossy encyclopedia

#22

Post by blitzar » Tue, Sep 02, 2025, 10:10 AM UTC

haktan wrote: post #19

It's also like someone who knows lots of facts but bad at remembering where they exactly learned it from.

and frequently confuses them with other facts they know

Re: An LLM is a lossy encyclopedia

#23

Post by smeeth » Tue, Sep 02, 2025, 10:11 AM UTC

As far as analogies go I prefer approximate database

Re: An LLM is a lossy encyclopedia

#24

Post by HeckFeck » Tue, Sep 02, 2025, 10:12 AM UTC

haktan wrote: post #19

It's also like someone who knows lots of facts but bad at remembering where they exactly learned it from.

And likely when mid-story to invent parts of the story to fill the gaps, rather than admit it is wrong.

Maybe the LLMs aren't so different from us.

Re: An LLM is a lossy encyclopedia

#25

Post by mr_toad » Tue, Sep 02, 2025, 10:12 AM UTC

That’s not the title of the article (granted, it doesn’t have one), and the author calls the analogy “questionable”.

Re: An LLM is a lossy encyclopedia

#26

Post by simonw » Tue, Sep 02, 2025, 10:13 AM UTC

feverzsj wrote: post #20

So, it's basically useless or even harmful.

Yes, if you try to use it as if it was an actual lossless encyclopedia.

One of the reasons I like this analogy is that it hints at the fact that you need to use them in a different way - you shouldn't be looking up specific facts in an unassisted LLM outside of things that even lossy compression would capture (like the capital cities of countries).

Re: An LLM is a lossy encyclopedia

#27

Post by Svip » Tue, Sep 02, 2025, 10:17 AM UTC

mr_toad wrote: post #25

That’s not the title of the article (granted, it doesn’t have one), and the author calls the analogy “questionable”.

In the HTML's -tag, it's called "Lossy encyclopedia".

Re: An LLM is a lossy encyclopedia

#28

Post by simonw » Tue, Sep 02, 2025, 10:19 AM UTC

mr_toad wrote: post #25

That’s not the title of the article (granted, it doesn’t have one), and the author calls the analogy “questionable”.

Most analogies are questionable in my experience - I find calling something a "questionable analogy" makes it a tiny bit less likely that people will pick it apart with thousands of reasons it's not an exact match for what it's describing.

Re: An LLM is a lossy encyclopedia

#29

Post by simonw » Tue, Sep 02, 2025, 10:26 AM UTC

thw_9a83c wrote: post #9

Yes, LLM is a lossy encyclopedia with a human-language answering interface. This has some benefits, mostly in terms of convenience. You don't have to browse or read through so many pages of a real encyclopedia to get a quick answer. However, there is also a clear downside. Currently, LLM is unable to judge if your question is formulated incorrectly or if your question opens up more questions that should be answered f…

I don't think that's universally true with the new models - I've seen Claude 4 and GPT-5 ask for clarification on questions with obvious gaps.

With GPT-5 I sometimes see it spot a question that needs clarifying in its thinking trace, then pick the most likely answer, then spit out an answer later that says "assuming you meant X ..." - I've even had it provide an answer in two sections for each branch of a clear ambiguity.

Re: An LLM is a lossy encyclopedia

#30

Post by latexr » Tue, Sep 02, 2025, 10:27 AM UTC

A lossy encyclopaedia should be missing information and be obvious about it, not making it up without your knowledge and changing the answer every time.

When you have a lossy piece of media, such as a compressed sound or image file, you can always see the resemblance to the original and note the degradation as it happens. You never have a clear JPEG of a lamp, compress it, and get a clear image of the Milky Way, then reopen the image and get a clear image of a pile of dirt.

Furthermore, an encyclopaedia is something you can reference and learn from without a goal, it allows you to peruse information you have no concept of. Not so with LLMs, which you have to query to get an answer.