data + plural = number data - plural = research king - crown = (didn't work... crown gets circled in red) king - princess = emperor king - queen = kingdom queen - king = worker king + queen = queen + king = kingdom boy + age = (didn't work... boy gets circled in red) man - age = woman woman - age = newswoman woman + age = adult female body (tied with man) girl + age = female child girl + old = female child The other…
For fun, I pasted these into ChatGPT o4-mini-high and asked it for an opinion: data + plural = datasets data - plural = datum king - crown = ruler king - princess = man king - queen = prince queen - king = woman king + queen = royalty boy + age = man man - age = boy woman - age = girl woman + age = elderly woman girl + age = woman girl + old = grandmother The results are surprisingly good, I don't think I could've do…
> The results are surprisingly good, I don't think I could've done better as a human
To clarify some of the issues:
data is both singular and plural, being a mass noun[0,1]. Datum is something you'll find in the dictionary, but not common in use[2]. The dictionary lags actual definitions. I mean words only mean what we collectively agree they mean (dictionary definitely helps with that but we also invent words all the time -- i.e. slang). I see how this one could trick up a human, feeling the need to change the output and would likely consult a dictionary but I don't think that's a fair comparison here as LLMs don't have these same biases.
King - crown really seems like it should be something like "man" or "person". The crown is the manifestation of the ruling power. We still use phrases like "heavy is the head that wears the crown" in reference to general leaders, not just monarchs.
king - princess I honestly don't know what to expect. Man is technically gender neutral so I'll take this one.
king - queen I would expect similar outputs to the previous one. Don't quite agree here.
queen - king I get why is removing royalty but given the previous (two) results I think is showing a weird gender bias. Remember that queen is something like (woman + crown) and king is akin to (man + crown). So subtracting should be woman - man.
The others I agree with. These were actually done because I was quite surprised at the results and was thinking about the aforementioned gender bias.
> But keep in mind that this doesn't do embedding math like OP!
After the embedding sub-network is your main transformer network. The purpose of this network is to perform embedding math! It is just that the goal is to do significantly more complicated math. Remember, these are learnable mappings (see Optimal Transport). We're just breaking it down into their two main intermediate mappings. But the embeddings still end up being a bottleneck. It is your literal gateway from words to numbers.
[0] https://en.wikipedia.org/wiki/Mass_noun
[1] https://www.merriam-webster.com/dictionary/data
[2] https://www.sciotoanalysis.com/news/2023/1/18/this-data-or-t...
[3] https://arxiv.org/abs/2005.14165
[4] https://arxiv.org/abs/2303.08774
[4] https://www.lesswrong.com/posts/3duR8CrvcHywrnhLo/how-does-g...