Building a language translator from scratch with deep learning
blog.floydhub.com
Building a language translator from scratch with deep learning
1–10 of 38 posts
Re: Building a language translator from scratch with deep learning
#2This is very cool. One thing I wonder about though is whether small companies will be able to compete with large ones like Google in ML in the future. One reason Google's translator is better is because they have way more data. In the past they digitized tons of books so they have an excellent dataset that has been translated by professional, human translators. This data collection is effectively cross-subsidized by Google's primary business: advertising.
Since most competitors to Google offerings aren't going to have a hugely profitable core business with which to fund all the data collection and normalization that goes into building a high quality ML system, the future for poorly capitalized competitors to compete seems bleak to me. This seems to support some of the growing rumblings about enforcing antitrust laws against the large tech companies.
Edit: better, not bigger.
Re: Building a language translator from scratch with deep learning
#3This is very cool. One thing I wonder about though is whether small companies will be able to compete with large ones like Google in ML in the future. One reason Google's translator is better is because they have way more data. In the past they digitized tons of books so they have an excellent dataset that has been translated by professional, human translators. This data collection is effectively cross-subsidized by…
Access to parallel corpora is a limiting factor in general. A good way to train a language translator is to use an open source dataset (several here http://opus.nlpl.eu/) to train a base model, and then fine-tune it with a smaller dataset specific to your domain.
In this case, the author claims pretty good accuracy, almost on par with Google Brain's!
On my test set of 3,000 sentences, the translator obtained a BLEU score of 0.39. This score is the benchmark scoring system used in machine translation, and the current best I could find in English to French is around 0.42 (set by some smart folks as Google Brain). So, not bad.Re: Building a language translator from scratch with deep learning
#4This is very cool. One thing I wonder about though is whether small companies will be able to compete with large ones like Google in ML in the future. One reason Google's translator is better is because they have way more data. In the past they digitized tons of books so they have an excellent dataset that has been translated by professional, human translators. This data collection is effectively cross-subsidized by…
Access to parallel corpora is a limiting factor in general. A good way to train a language translator is to use an open source dataset (several here http://opus.nlpl.eu/ ) to train a base model, and then fine-tune it with a smaller dataset specific to your domain. In this case, the author claims pretty good accuracy, almost on par with Google Brain's! On my test set of 3,000 sentences, the translator obtained a BLEU…
Wow, missed that part when I read it. Pretty incredible that using open source data you can outperform the state-of-the-art machine translators of a few years ago.
Re: Building a language translator from scratch with deep learning
#5Earlier quoted context omitted.
Access to parallel corpora is a limiting factor in general. A good way to train a language translator is to use an open source dataset (several here http://opus.nlpl.eu/ ) to train a base model, and then fine-tune it with a smaller dataset specific to your domain. In this case, the author claims pretty good accuracy, almost on par with Google Brain's! On my test set of 3,000 sentences, the translator obtained a BLEU…
Wow, missed that part when I read it. Pretty incredible that using open source data you can outperform the state-of-the-art machine translators of a few years ago.
For a historical perspective check out stanford's nlp course: https://youtu.be/IxQtK2SjWWM?t=1267
Deep learning only started beating tradition methods in 2016!
Re: Building a language translator from scratch with deep learning
#6This is very cool. One thing I wonder about though is whether small companies will be able to compete with large ones like Google in ML in the future. One reason Google's translator is better is because they have way more data. In the past they digitized tons of books so they have an excellent dataset that has been translated by professional, human translators. This data collection is effectively cross-subsidized by…
Access to parallel corpora is a limiting factor in general. A good way to train a language translator is to use an open source dataset (several here http://opus.nlpl.eu/ ) to train a base model, and then fine-tune it with a smaller dataset specific to your domain. In this case, the author claims pretty good accuracy, almost on par with Google Brain's! On my test set of 3,000 sentences, the translator obtained a BLEU…
Yes, it shouldn't have been far off Google as the model in the article is Google's itself. In fact you see in some of the examples how Google Translate's output and this model are almost exactly the same.
Pretraining is a very cool idea, I have also seen some good results with pretrained embeddings after doing language modelling. Fast ai discusses this and I think even has some pretrained embeddings available in their library!
Re: Building a language translator from scratch with deep learning
#7This is very cool. One thing I wonder about though is whether small companies will be able to compete with large ones like Google in ML in the future. One reason Google's translator is better is because they have way more data. In the past they digitized tons of books so they have an excellent dataset that has been translated by professional, human translators. This data collection is effectively cross-subsidized by…
DeepL had a lot of good press when it came out last year. Some saying it was better than Google.
Re: Building a language translator from scratch with deep learning
#8Despite 'The unreasonable effectiveness of RNNs', it's seeming CNNs and the solely attention-based models are managing to perform the same tasks, but with better results and faster speed!
Re: Building a language translator from scratch with deep learning
#9This is very cool. One thing I wonder about though is whether small companies will be able to compete with large ones like Google in ML in the future. One reason Google's translator is better is because they have way more data. In the past they digitized tons of books so they have an excellent dataset that has been translated by professional, human translators. This data collection is effectively cross-subsidized by…
The value of large corpora for translation may be diminishing.. In particular, Facebook have achieved impressive results using unsupervised ML for translation: https://code.fb.com/ai-research/unsupervised-machine-transla...
The basic idea is to use word vector embeddings to build a sourcetarget dictionary, then combine this with a language recognition model to iteratively bootstrap a set of sourcetarget training examples for use with a conventional ML approach.
Re: Building a language translator from scratch with deep learning
#10This is very cool. One thing I wonder about though is whether small companies will be able to compete with large ones like Google in ML in the future. One reason Google's translator is better is because they have way more data. In the past they digitized tons of books so they have an excellent dataset that has been translated by professional, human translators. This data collection is effectively cross-subsidized by…
Another perspective in similar veins would be the rise of AutoML. Given its absurdly high computational cost, I'd think only enterprises with massive computational power at their disposal would be able to use it.