Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Differences between the word2vec paper and its implementation

github.com

Post reply on HN
Print view View original

21–30 of 156 posts

Re: Differences between the word2vec paper and its implementation

#21

Post by chriskanan »

newen wrote: post #2

This kind of things happens all the time in academia. The authors are either constrained by space due to paper limitations or they are too lazy to explain all the little details that go into the algorithm. I used to do research in computer vision a few years ago and it used to be that people won't publish their code and they purposely won't put in all of the details of the algorithm in the paper. Many of those algori…

Completely agree. The situation today is far better than a decade ago, with code releases for machine learning and computer vision papers being much more common than before.

I try to make my students release polished and easy to use code, but it can sometimes fall through the cracks due to deadlines, etc. Many projects are the output of a single PhD student.

Re: Differences between the word2vec paper and its implementation

#22

Post by billconan »

liuliu wrote: post #13
billconan wrote: post #6

For the past week I have been frustrated by an opensource code of a deep learning paper. This type of things are so common in academia. The particular code I looked at has missing documentation, hardcoded local paths, broken dataset download links and broken pretrained model download links. I have to fix bugs before the code can run. I'm very curious how did the author run that code with the bugs. I call them insince…

Maintaining a pipeline that works takes effort that can be spent somewhere else. As long as it can be reproduced with a little effort from me, I really don't mind. I would rather they spend time to try a lot of different things, build intuitions, and search better backbone structures than maintaining a CI system so that every time their code updates can train imageNet to That should be left to TensorFlow model zoo or…

I'm not asking for much, but the code should be runnable at least.

This is why I think jupyter notebook and things alike are so important as the future publishing media. Reproducibility is very important.

Re: Differences between the word2vec paper and its implementation

#23

Post by toast0 »

bollu wrote: post #5
newen wrote: post #2

This kind of things happens all the time in academia. The authors are either constrained by space due to paper limitations or they are too lazy to explain all the little details that go into the algorithm. I used to do research in computer vision a few years ago and it used to be that people won't publish their code and they purposely won't put in all of the details of the algorithm in the paper. Many of those algori…

Is this really that common? That's disheartening, I want to spend time in academia but experiences like this are sucking the fun out for me...

I tried to make use of some public audio research and it was pretty bad. There was an audio comprehensibility competition a few years ago. Some of the papers submitted are still around, as well as the summary paper describing the results. But many papers are hard to find, and those that claimed to have source code available are hard to find --- i was able to get matlab sources for a few algorithms, but they somehow work on the example files, but mostly crash on my files.

It's a shame because I understand the idea of the paper, and have an excellent place to apply it, but I lack the DSP background, so I can't really rebuild the code from scratch -- so the work is not able to be used.

Re: Differences between the word2vec paper and its implementation

#25

Post by bollu »

DannyBee wrote: post #17

Speaking as someone who has read about 40 years of papers in compiler optimization, it's very interesting. In the early days, there were fairly exact algorithms that worked as described, and were implemented as described, but were pseudocoded in papers. Where the pseudocode differed from implementation, differences were described in great detail (IE they may say an array can be shared but isn't to make the pseudocode…

As someone who works on compiler optimisations, I'd love to get a link the Rice compiler's source code --- do you happen to have a link?

Re: Differences between the word2vec paper and its implementation

#27

Post by utopcell »

I think it very unfair to the original set of word2vec papers to be talking about 'academic dishonesty'. This is a case of a user that has little to no experience with neural networks. There are a ton of articles describing the need for random initialization [1][2]. In fact, if one spends a few seconds thinking about it, the need is evident. Without it, the NN cannot perform symmetry breaking: If inputs are set to zero, all neurons will perform the same calculations, rendering the network useless.

[1] google: "neural networks vector initialization" [2] http://deeplearning.ai/ai-notes/initialization/

Re: Differences between the word2vec paper and its implementation

#28

Post by not2b »

nickpsecurity wrote: post #18
billconan wrote: post #6

For the past week I have been frustrated by an opensource code of a deep learning paper. This type of things are so common in academia. The particular code I looked at has missing documentation, hardcoded local paths, broken dataset download links and broken pretrained model download links. I have to fix bugs before the code can run. I'm very curious how did the author run that code with the bugs. I call them insince…

One explained to me here that the problem is they're mainly rewarded by funding agencies for how many papers they publish. That's why quality went down in general. Then, since funding agencies aren't rewarding code, they're either not encouraged to get it right or actively discouraged from doing it to focus on next paper. This sad situation is both bad for open source and science. Bad for science since inaccurate res…

Then some other academic can get credit for publishing a rigorous paper describing the actual algorithm, explaining why the original paper is wrong (without speculating on the motive, just the facts), and why the approach taken in the code is in fact better. Bonus points for describing a further improvements, with working code.

Re: Differences between the word2vec paper and its implementation

#29

Post by bollu »

utopcell wrote: post #27

I think it very unfair to the original set of word2vec papers to be talking about 'academic dishonesty'. This is a case of a user that has little to no experience with neural networks. There are a ton of articles describing the need for random initialization [1][2]. In fact, if one spends a few seconds thinking about it, the need is evident. Without it, the NN cannot perform symmetry breaking: If inputs are set to ze…

That's hardly the point of the article --- the actual paper does not describe the use of two separate vectors for each word. The initialization was an interesting tidbit.

Re: Differences between the word2vec paper and its implementation

#30

Post by throwaway287391 »

billconan wrote: post #6

For the past week I have been frustrated by an opensource code of a deep learning paper. This type of things are so common in academia. The particular code I looked at has missing documentation, hardcoded local paths, broken dataset download links and broken pretrained model download links. I have to fix bugs before the code can run. I'm very curious how did the author run that code with the bugs. I call them insince…

This sort of thing is aggravating to read. Frankly it comes off as really entitled. As researchers, the expectation is now that we not only have to do the research and write a paper like the good old days, but we have to release the code too. Okay, fine. But now that's not enough either -- the code has to be well-documented and clean. Ugh, alright, fine -- it's going to take me a few extra weeks of not doing research, but I'll clean up all the code, rerun experiments to make sure it all still works like before, and add a bunch of documentation. But no, still not good enough -- it has to run at the press of a button in your particular programming environment. If we don't know how to write a script (or couldn't be bothered to spend the time writing one) to check that the data is on disk and, if not, crawl a website to download some huge dataset in one click, test our code on your OS, your CPU/GPU/TPU/..., etc., we were being "insincere" with our open-sourcing efforts.
Post reply on HN