Live data from Hacker News

From tokens to thoughts: How LLMs and humans trade compression for meaning

arxiv.org

21–27 of 27 posts

Re: From tokens to thoughts: How LLMs and humans trade compression for meaning

#21

Earlier quoted context omitted.

Not sure how they're fundamentally different. What do you mean?

Think about the work of localizing a joke that relies on wordplay or similar sounding words to be funny. Or simply how words rhyme Try explaining why tough and rough rhyme but bough doesn't You know? Language has a ton of idiosyncrasies.

ChatGPT is horrible at producing Dutch rhymes (for Sinterklaas poems) until you realize that the words it comes up with do rhyme when translated to English.

Re: From tokens to thoughts: How LLMs and humans trade compression for meaning

#23

Earlier quoted context omitted.

And that is just in English Other languages have similar but fundamentally different oddities which do not translate cleanly

Not sure how they're fundamentally different. What do you mean?

As most languages allow expressions of algorithms, they are all Turing complete and, thus, are not fundamentally different. The complexity of expressions of some concepts is different, though.

My favorite thing is a "square." I put that name to an enumeration that allows me to compare and contrast things with two different qualities expressed by two extremes.

One such square is "One can (not) do (not do) something." Both "not"'s can be present and absent, just like a truth table.

"One can do something", "one can not do something", "one can do not do something" and, finally, "one can not help but do something."

Why should we use "help but" instead of "do not"?

While this does not preclude one from enumerating possibilities thinking in English, it makes that enumeration harder than it can be in other languages. For example, in Russian the "square" is expressible directly.

Also, "help but" is not shorter than "do not," it is longer. Useful idioms usually expressed in shorter forms, thus, apparently, "one can not help but do something" is considered by Englishmen as not useful.

Re: From tokens to thoughts: How LLMs and humans trade compression for meaning

#25

Earlier quoted context omitted.

Not sure how they're fundamentally different. What do you mean?

Think about the work of localizing a joke that relies on wordplay or similar sounding words to be funny. Or simply how words rhyme Try explaining why tough and rough rhyme but bough doesn't You know? Language has a ton of idiosyncrasies.

Right but I wouldn't call those things fundamentally different. That's just having different words; the categories of idiosyncrasies are still the same.

Re: From tokens to thoughts: How LLMs and humans trade compression for meaning

#26
post #19

Earlier quoted context omitted.

Think about the work of localizing a joke that relies on wordplay or similar sounding words to be funny. Or simply how words rhyme Try explaining why tough and rough rhyme but bough doesn't You know? Language has a ton of idiosyncrasies.

To make it more concrete - here's an example in Chinese: https://en.wikipedia.org/wiki/Grass_Mud_Horse

> 2009, renowned artist Ai Weiwei published an image of himself nude with only a 'Caonima' hiding his genitals, with a caption "草泥马挡中央" (cǎonímǎ dǎng zhōngyāng; 'a Grass Mud Horse covering the center'. One interpretation of the caption is: "fuck your mother, Communist Party Central Committee"). Political observers speculated that the photo may have contributed to Ai's arrest in 2011 by angering Chinese Communist Party hardliners.

How did I never hear about this detail??

Re: From tokens to thoughts: How LLMs and humans trade compression for meaning

#27

This paper is interesting, but ultimately it's just restating that LLMs are statistical tools and not cognitive systems. The information-theoretic framing doesn’t really change that.

my take from the paper is that the authors overstate the following:

LLMs cannot zoom in and out of certain details, while we humans subconsciously zoom in and out of context and even go through conceptual spaces in the very hermetic sense of it. LLMs do not. they work on the present context, which kickstarts them into expounding likeliness in the same contextual space.

Post reply on HN