11–20 of 25 posts
I have a feeling the answer is “no”, but can anyone comment on whether or not this could be used to decode the utterances of other animals, such as whales?
The previous paper they mention explains the core insight that makes unsupervised translation possible: https://arxiv.org/abs/1710.04087 The original paper didn't receive the attention I thought it would, but I continue to think this is a fascinating result which has deep implications for machine learning and for linguistics.
These word embeddings keep on yielding all kinds of amazing benefits. Is there any kind of explainability research to help people understand them better in terms of human psychology?
There has been much in the way of discussion for, and concern about (for example, the Deconstructionist movement), these ideas, for the past 60 years or so. And a bit of practical exploration in the field of child development.
Interestingly, the fact that embeddings between languages seem to share some common shapes (per the linked paper in this thread), would seem to suggest that A) Fundamentally, most languages have the same deep structure, whether through coincidence or common evolutionary root. or B) The brain has a hard-wired structure for language, evolved alongside the development of language itself. The Chomskian Language Acquisition Device. We're not born tabula rasa, we've got some hardcoding indicating how we're going to understand things
Or a little of A, a little of B maybe, as it does end up being a boostrapping problem.
In one of my previous companies we used this technique to hire pair of translators: we gave translator pairs such task, and hired pair which reconstructed original text more closely.
So, they built translation system using bilingual dictionaries, then asked it to translate from English to Urdu, and then back to English, and minimized loss between original and double-translated English. In one of my previous companies we used this technique to hire pair of translators: we gave translator pairs such task, and hired pair which reconstructed original text more closely.
I have a feeling the answer is “no”, but can anyone comment on whether or not this could be used to decode the utterances of other animals, such as whales?
You would need a bilingual dictionary as the first step of the process, so I'm afraid that this will not be possible.
So, they built translation system using bilingual dictionaries, then asked it to translate from English to Urdu, and then back to English, and minimized loss between original and double-translated English. In one of my previous companies we used this technique to hire pair of translators: we gave translator pairs such task, and hired pair which reconstructed original text more closely.
No, they didn't start with dictionaries, or any other parallel corpora; they learned the word by word translations as well from monolingual corpora, by finding alignments between monolingual word embeddings in the target languages.
So, they built translation system using bilingual dictionaries, then asked it to translate from English to Urdu, and then back to English, and minimized loss between original and double-translated English. In one of my previous companies we used this technique to hire pair of translators: we gave translator pairs such task, and hired pair which reconstructed original text more closely.
2. rotate embeddings space of two languages for optimal alignment, assuming frequency and neighborhoods of word embeddings are more-less the same in any language
3. iteratively minimize difference in bidirectional translations
I have a feeling the answer is “no”, but can anyone comment on whether or not this could be used to decode the utterances of other animals, such as whales?
To translate whale song, you need data that corresponds to what the whales are singing about.
So, they built translation system using bilingual dictionaries, then asked it to translate from English to Urdu, and then back to English, and minimized loss between original and double-translated English. In one of my previous companies we used this technique to hire pair of translators: we gave translator pairs such task, and hired pair which reconstructed original text more closely.
No, they didn't start with dictionaries, or any other parallel corpora; they learned the word by word translations as well from monolingual corpora, by finding alignments between monolingual word embeddings in the target languages.
The previous paper they mention explains the core insight that makes unsupervised translation possible: https://arxiv.org/abs/1710.04087 The original paper didn't receive the attention I thought it would, but I continue to think this is a fascinating result which has deep implications for machine learning and for linguistics.
Normally this technique wouldn't be useful, because it's overfitting a specific training set. (If you make space X as similar as possible to space Y, then this mapping from X to Y is only useful for X to Y – it can't generalize to other situations, which is often the goal of an ML model.)
But since the task is "Translate from English to Italian," and since all languages have similar embedding structures (Zipf mystery), overfitting is exactly what we want: we want, for any given English phrase, to find the closest-fitting mapping to a corresponding Italian phrase.
The more I learn about ML and data mining, the more I'm astounded by how clever many of the techniques are, and how much artistry is involved. You have to be clever to make a certain model perform well in a certain domain. If you want to make a stock trading bot, you can't randomly subdivide stock market data into e.g. 70% training data and 30% test data, because the data is ordered by time. You have to use the past 3 years of stock market data as a training set, and validate it against the subsequent 1 year of market data.
I really like ML because the techniques applicable for training a stock market bot seem unrelated to the algorithms for doing unsupervised machine translation, which differ from how to model credit fraud, which are no doubt different from how to build a dota 2 bot. :)