21–25 of 25 posts
Wouldn't the method work better with n-gram embeddings, where n=3 or 4?
The previous paper they mention explains the core insight that makes unsupervised translation possible: https://arxiv.org/abs/1710.04087 The original paper didn't receive the attention I thought it would, but I continue to think this is a fascinating result which has deep implications for machine learning and for linguistics.
This is genius: https://imgur.com/a/1aRZ3sI Normally this technique wouldn't be useful, because it's overfitting a specific training set. (If you make space X as similar as possible to space Y, then this mapping from X to Y is only useful for X to Y – it can't generalize to other situations, which is often the goal of an ML model.) But since the task is "Translate from English to Italian," and since all languages hav…
All of this seems to be using word embeddings, but most languages don't have all that many words. You're effectively trying to train with just a few thousand data points, and will quickly overfit. Wouldn't the method work better with n-gram embeddings, where n=3 or 4?
And since you are simply learning a rotation matrix, there is no risk of overfitting.
Looks like promising research. I will have to read the actual paper later as opposed to just the blog post. One thing I would like to say is my qualm with the obsession with unsupervised learning as the quintessential technique. Unsupervised != no human input. Having the benefit of knowing about word embeddings is something inherently built into this system by the human designers and something that supervised learnin…
> One thing I would like to say is my qualm with the obsession with unsupervised learning as the quintessential technique. Unsupervised != no human input. I'm not sure I follow the qualm you are trying to get across. Are you saying you disagree with the term 'unsupervised' because unsupervised algorithms still bake in human assumptions (like a human-designed word embedding model) so that's essentially still supervisi…
Where I disagree with you is that the obsession is purely driven from "results-per-dollar-invested", at least in the academic world. That being said, unsupervised learning is a great tool and definitely worthy of research.
To summarize, my comment was completely tangential of this paper (the authors make no such claims). It was more of a stream of consciousness comment that arose because I envisioned someone reading the paper and saying "see!, unsupervised learning leads to real understanding, no humans needed!"