Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

A novel approach to neural machine translation

code.facebook.com

Post reply on HN
Print view View original

21–30 of 45 posts

Re: A novel approach to neural machine translation

#21

Post by laGrenouille »

jorgemf wrote: post #17
forgotmyhnacc wrote: post #3

I really like that Facebook open sources both code and model along with the paper. Most companies don't: e.g. Google, deepmind, Baidu.

There is a github link in the article with both things: https://github.com/facebookresearch/fairseq Google and Deepmind released a lot of stuff, I don't feel I have the right to complain about it.

I feel the right to complain anytime Google, Baidu, or Deepmind want to publish results from their models in peer-reviewed forums without offering the models for public scrutiny. If they want to keep the models internal, that's fine, and if they want to be taken seriously in academia that is also fine, but they can't have it both ways.

Re: A novel approach to neural machine translation

#24

Post by albertzeyer »

forgotmyhnacc wrote: post #3

I really like that Facebook open sources both code and model along with the paper. Most companies don't: e.g. Google, deepmind, Baidu.

Google has released some framework for translation: https://github.com/google/seq2seq/

DeepMind has also released some framework which also has all the building blocks for translation: https://github.com/deepmind/sonnet

Re: A novel approach to neural machine translation

#25

Post by skybrian »

pwaivers wrote: post #8

As far I understood it, Facebook put lots of research into optimizing a certain type of neural network (CNN), while everyone else is using another type called RNN. Up until now, CNN was faster but less accurate. However FB has progressed CNN to the point where it can compete in accuracy, particularly in speech recognition. And most importantly, they are releasing the source code and papers. Does that sound right? Can…

Not an expert, but as I understand it, common practice (everywhere, not just at Facebook) is to use CNN for understanding images and other kinds of non-sequential data. RNN are commonly used for handling text and other kinds of sequential data.

They showed how to use a CNN with text to get a speed boost, even though that's not how it's normally been done.

Re: A novel approach to neural machine translation

#26

Post by halflings »

laGrenouille wrote: post #21
jorgemf wrote: post #17

Earlier quoted context omitted.

There is a github link in the article with both things: https://github.com/facebookresearch/fairseq Google and Deepmind released a lot of stuff, I don't feel I have the right to complain about it.

I feel the right to complain anytime Google, Baidu, or Deepmind want to publish results from their models in peer-reviewed forums without offering the models for public scrutiny. If they want to keep the models internal, that's fine, and if they want to be taken seriously in academia that is also fine, but they can't have it both ways.

Ah, yes, Google are not taken seriously in academia because they are not releasing the code source to their models. Google is usually the largest contributor to most ML conferences (NIPS, ICML, etc.)

Most papers are not about implementation and more about the concepts or proofs. They are rather straightforward to reimplement, and I don't think anybody is accusing them of faking their results.

Re: A novel approach to neural machine translation

#29

Post by rryan »

laGrenouille wrote: post #21
jorgemf wrote: post #17

Earlier quoted context omitted.

There is a github link in the article with both things: https://github.com/facebookresearch/fairseq Google and Deepmind released a lot of stuff, I don't feel I have the right to complain about it.

I feel the right to complain anytime Google, Baidu, or Deepmind want to publish results from their models in peer-reviewed forums without offering the models for public scrutiny. If they want to keep the models internal, that's fine, and if they want to be taken seriously in academia that is also fine, but they can't have it both ways.

> I feel the right to complain anytime

Most people on the Internet do.

Re: A novel approach to neural machine translation

#30

Post by baristaGeek »

mabbo wrote: post #14
pwaivers wrote: post #8

As far I understood it, Facebook put lots of research into optimizing a certain type of neural network (CNN), while everyone else is using another type called RNN. Up until now, CNN was faster but less accurate. However FB has progressed CNN to the point where it can compete in accuracy, particularly in speech recognition. And most importantly, they are releasing the source code and papers. Does that sound right? Can…

I'll give it a shot. Traditional Neural Networks worked like this: You have k inputs to a layer, and j outputs, so you have O(k * j) parameters, effectively multiplying the inputs by the parameter to get the outputs. And if you have lots of inputs to each layer, and lots of layers, you have a lot of parameters. Too many parameters = overfitting to your training data pretty quickly. But you want big networks, ideally,…

That's a great explanation.

Let me add this though:

Artificial neural networks were proposed to compute the probability of a sequence of words occurring; however, RNNs were the next step in Natural Language Processing since they allow variable-length sequences to be received as an input contrary to the previously proposed architecture.

However a simple RNN architecture didn't allow for long -term dependencies to be captured (that is, use statistical modeling to predict a word sequence on a part of a text that is based on an idea previously developed on the corpus). So two kinds of fancy RNN architectures were developed to tackle this problem: GRUs and LSTMs. Production systems are already implementing these architectures and they are yielding pretty accurate results.

But now Facebook researchers are proposing using CNNs for this task because this architecture can take more advantage of GPU parallelism.

Post reply on HN