31–40 of 138 posts
I've been trying to do this with just word2vec, instead of throwing an LLM, since you just need to find a word with the appropriate distances optimized. https://github.com/captn3m0/ideas?tab=readme-ov-file#codenam...
Imagine the energy savings if more people didn’t just automatically reach for LLMs for their pet projects.
I've intuitively felt that this general class of task is what these LLMs are absolutely best at. I'm not an expert on these things, but isn't this thanks to word embeddings and how words are mapped into high dimensional vector space within the model? I would imagine that because every word is mapped this way, finding a word that exists in the same area as mail, lawyer, log, and line in some vector space would be triv…
Earlier quoted context omitted.
> But the guesser model connects Paper and Log because "paper is made from logs" (reaching the conclusion through a different meaning of Log) No, it doesn't. It reaches the conclusion because of vector similarity (simplified explanation): these explanations are post-hoc .
Sorry, I’m uninformed. Do you mean thaw the explanation could be completely unrelated to the actual “reason”
Video series on the topic: https://www.3blue1brown.com/topics/neural-networks
Which is to say that “why” it gives those answers is because its statistically likely within its training data that when there are the words, “why did you connect line and log with paper” the text which follows could be “logs are made of wood and lines are in paper.” But that is not the specific relation of the 3 words in the model itself, which is just a complex vector space.
Earlier quoted context omitted.
This is the take I thought I'd have, but in the last example, the guesser model reaches the correct conclusion using a different reasoning than the clue giver model. The clue giver justifies the link of Paper and Log as "written records", and between Paper and Line as "lines of text". But the guesser model connects Paper and Log because "paper is made from logs" (reaching the conclusion through a different meaning of…
> But the guesser model connects Paper and Log because "paper is made from logs" (reaching the conclusion through a different meaning of Log) No, it doesn't. It reaches the conclusion because of vector similarity (simplified explanation): these explanations are post-hoc .
The best available evidence suggests this is also true of any explanations a human gives for their own behaviour; nevertheless we generally accept those at face value.
Earlier quoted context omitted.
> But the guesser model connects Paper and Log because "paper is made from logs" (reaching the conclusion through a different meaning of Log) No, it doesn't. It reaches the conclusion because of vector similarity (simplified explanation): these explanations are post-hoc .
> these explanations are post-hoc. The best available evidence suggests this is also true of any explanations a human gives for their own behaviour; nevertheless we generally accept those at face value.
Earlier quoted context omitted.
Sorry, I’m uninformed. Do you mean thaw the explanation could be completely unrelated to the actual “reason”
Yes, the reason is that the model assigns words positions in an ever-changing vector space and evaluates relation by their correspondence in that space—the reply it gives is also a certain index of that space, with the “why” in the question giving it the weight of producing an “answer.” Video series on the topic: https://www.3blue1brown.com/topics/neural-networks Which is to say that “why” it gives those answers is b…
I did this with Claude over the holidays. Putting Claude in the role as a guesser and comparing the guess to another experience human player. It turns out they both matched each other.
That's a nice experiment! I think codenames could definietly be an evaluation method for LLMs.
I've been trying to do this with just word2vec, instead of throwing an LLM, since you just need to find a word with the appropriate distances optimized. https://github.com/captn3m0/ideas?tab=readme-ov-file#codenam...