From tokens to thoughts: How LLMs and humans trade compression for meaning
1–10 of 27 posts
Re: From tokens to thoughts: How LLMs and humans trade compression for meaning
#2>> For each LLM, we extract static, token-level embeddings from its input embedding layer (the ‘E‘matrix). This choice aligns our analysis with the context-free nature of stimuli typical in human
categorization experiments, ensuring a comparable representational basis.
They're analyzing input embedding models, not LLMs. I'm not sure how the authors justify making claims about the inner workings of LLMs when they haven't actually computed a forward pass. The EMatrix is not an LLM, its a lookup table.
Just to highlight the ridiculousness of this research, no attention was computed! Not a single dot product between keys and queries. All of their conclusions are drawn from the output of an embedding lookup table.
The figure showing their alignment score correlated with model size is particularly egregious. Model size is meaningless when you never activate any model parameters. If Bert is outperforming Qwen and Gemma something is wrong with your methodology.
Re: From tokens to thoughts: How LLMs and humans trade compression for meaning
#3>> For each LLM, we extract static, token-level embeddings from its input embedding layer (the ‘E‘matrix). This choice aligns our analysis with the context-free nature of stimuli typical in human categorization experiments, ensuring a comparable representational basis. They're analyzing input embedding models, not LLMs. I'm not sure how the authors justify making claims about the inner workings of LLMs when they have…
Note that the token embeddings are also trained, therefore their values do give some hints on how a model is organizing information.
They used token embeddings directly and not intermediate representations because the latter depend on the specific sentence that the model is processing. Data on human judgment was however collected without any context surrounding each word, thus using the token embeddings seem to be the most fair comparison.
Otherwise, what sentence(s) would you have used to compute the intermediate representations? And how would you make sure that the results aren't biased by these sentences?
Re: From tokens to thoughts: How LLMs and humans trade compression for meaning
#4[deleted]
Re: From tokens to thoughts: How LLMs and humans trade compression for meaning
#5>> For each LLM, we extract static, token-level embeddings from its input embedding layer (the ‘E‘matrix). This choice aligns our analysis with the context-free nature of stimuli typical in human categorization experiments, ensuring a comparable representational basis. They're analyzing input embedding models, not LLMs. I'm not sure how the authors justify making claims about the inner workings of LLMs when they have…
It’s mind blowing LeCun is listed as one of the authors.
I would expect model size to correlate with alignment score because usually model sizes correlate with hidden dimension. But also opposite can be true - bigger models might shift more basic token classification logic into layers and hence embedding alignment can go down. Regardless feels like pretty useless research…
Re: From tokens to thoughts: How LLMs and humans trade compression for meaning
#6Am I the only one that is lost on how the calculations are made?
From what I can tell this is limited in scope to categorizing nouns (robin is a bird).
Re: From tokens to thoughts: How LLMs and humans trade compression for meaning
#7Open a bank account. Open your heart. Open a can. Open to new experiences.
Words are a tricky thing to handle.
Re: From tokens to thoughts: How LLMs and humans trade compression for meaning
#8Open a bank account. Open your heart. Open a can. Open to new experiences. Words are a tricky thing to handle.
OpenAI agrees
Re: From tokens to thoughts: How LLMs and humans trade compression for meaning
#9Open a bank account. Open your heart. Open a can. Open to new experiences. Words are a tricky thing to handle.
And models since BERT and ELMo capture polysemy!
Re: From tokens to thoughts: How LLMs and humans trade compression for meaning
#10Open a bank account. Open your heart. Open a can. Open to new experiences. Words are a tricky thing to handle.
And that is just in English
Other languages have similar but fundamentally different oddities which do not translate cleanly