Unsupervised machine translation
code.fb.com
Unsupervised machine translation
1–10 of 25 posts
Re: Unsupervised machine translation
#2Looks like promising research. I will have to read the actual paper later as opposed to just the blog post. One thing I would like to say is my qualm with the obsession with unsupervised learning as the quintessential technique. Unsupervised != no human input. Having the benefit of knowing about word embeddings is something inherently built into this system by the human designers and something that supervised learning does not have the benefit of when given original-translated pairs.
Re: Unsupervised machine translation
#3The previous paper they mention explains the core insight that makes unsupervised translation possible: https://arxiv.org/abs/1710.04087
The original paper didn't receive the attention I thought it would, but I continue to think this is a fascinating result which has deep implications for machine learning and for linguistics.
Re: Unsupervised machine translation
#4Looks like promising research. I will have to read the actual paper later as opposed to just the blog post. One thing I would like to say is my qualm with the obsession with unsupervised learning as the quintessential technique. Unsupervised != no human input. Having the benefit of knowing about word embeddings is something inherently built into this system by the human designers and something that supervised learnin…
> One thing I would like to say is my qualm with the obsession with unsupervised learning as the quintessential technique. Unsupervised != no human input.
I'm not sure I follow the qualm you are trying to get across. Are you saying you disagree with the term 'unsupervised' because unsupervised algorithms still bake in human assumptions (like a human-designed word embedding model) so that's essentially still supervision?
The obsession with 'unsupervised' learning as the quintessential technique is about getting better results for less money/effort. The premise is that we assume deep models tend to scale up in accuracy as training data size increases, so we always want larger datasets to increase accuracy. But creating labeled data takes a linear amount of human effort ($$$) as the dataset size increases. At a certain point, creating more labeled data to improve a model is not cost effective or maybe even impossible.
Unlabeled data can be acquired nearly for free in nearly unlimited quantities in many cases. So if we can use unlabeled data instead (even if it requires complex pre-processing like CBOW embedding models which essentially turn bits of the unlabeled data into it's own label) your final results-per-dollar-invested goes through the roof over supervised learning. That's the obsession. It's not about literally no supervision being involved in the process. It's about driving down the cost of data acquisition while driving up the percent of the world's available data you can use for training a model.
I apologize in advance if I'm missing the point you are making.
Re: Unsupervised machine translation
#5The previous paper they mention explains the core insight that makes unsupervised translation possible: https://arxiv.org/abs/1710.04087 The original paper didn't receive the attention I thought it would, but I continue to think this is a fascinating result which has deep implications for machine learning and for linguistics.
These word embeddings keep on yielding all kinds of amazing benefits. Is there any kind of explainability research to help people understand them better in terms of human psychology?
Re: Unsupervised machine translation
#6Looks like promising research. I will have to read the actual paper later as opposed to just the blog post. One thing I would like to say is my qualm with the obsession with unsupervised learning as the quintessential technique. Unsupervised != no human input. Having the benefit of knowing about word embeddings is something inherently built into this system by the human designers and something that supervised learnin…
Are you confusing unsupervised with model-free? Unsupervised/supervised doesn't say anything about the priors baked in to the model.
Re: Unsupervised machine translation
#7The previous paper they mention explains the core insight that makes unsupervised translation possible: https://arxiv.org/abs/1710.04087 The original paper didn't receive the attention I thought it would, but I continue to think this is a fascinating result which has deep implications for machine learning and for linguistics.
These word embeddings keep on yielding all kinds of amazing benefits. Is there any kind of explainability research to help people understand them better in terms of human psychology?
Word embeddings work because they reflect co-occurrences. I don't know whether that counts as an explanation in terms of psychology, but humans tend to put related things together. In a newspaper the articles aren't jumbled together, but there are sections on different topics, and in each section the articles are clearly delineated instead of mixing their sentences and each sentence represents a single unit instead of giving partial information on a dozen unrelated things.
It might seem obvious that things should be done that way, but if you consider servers hosting lots of different websites on the same physical machine, or data structures spread out over several memory allocations held together by pointers, it's clear that there are other possibilities. So it does seem to be specific to the way humans use language.
And because human language has this property of co-occurrences corresponding to relatedness in meaning, you can represent the meaning of a word by building a model that only predicts the probability that two words occur together.
Re: Unsupervised machine translation
#8Earlier quoted context omitted.
These word embeddings keep on yielding all kinds of amazing benefits. Is there any kind of explainability research to help people understand them better in terms of human psychology?
Word embeddings work because they reflect co-occurrences. I don't know whether that counts as an explanation in terms of psychology, but humans tend to put related things together. In a newspaper the articles aren't jumbled together, but there are sections on different topics, and in each section the articles are clearly delineated instead of mixing their sentences and each sentence represents a single unit instead o…
In linguistics, there's a classic principle "You shall know a word by the company it keeps" (Firth, J. R. 1957) - colocations (a sequence of words or terms that co-occur more often than would be expected by chance) are very informative about what a word means.
Re: Unsupervised machine translation
#9"We can improve upon this by making local edits using a language model that has been trained on lots of monolingual data to score sequences of words in such a way that fluent sentences score higher than ungrammatical or poorly constructed sentences."
Does anyone knows some references / corpus in English language for this?
Re: Unsupervised machine translation
#10I have a feeling the answer is “no”, but can anyone comment on whether or not this could be used to decode the utterances of other animals, such as whales?