Differences between the word2vec paper and its implementation • Hacker News Board

Re: Differences between the word2vec paper and its implementation

#41

Post by draw_down » Mon, Jun 03, 2019, 11:46 PM UTC

liquidify wrote: post #38

throwaway287391 wrote: post #30

Earlier quoted context omitted.

This sort of thing is aggravating to read. Frankly it comes off as really entitled. As researchers, the expectation is now that we not only have to do the research and write a paper like the good old days, but we have to release the code too. Okay, fine. But now that's not enough either -- the code has to be well-documented and clean. Ugh, alright, fine -- it's going to take me a few extra weeks of not doing research…

If there are bugs in your code that would have prevented you from being able to do what you claim to do, then to the readers of your paper, it seems more likely that you might not have performed the experimentation that you claim to.

I'd say that depends on the readers an awful lot.

Re: Differences between the word2vec paper and its implementation

#42

Post by Veedrac » Mon, Jun 03, 2019, 11:48 PM UTC

throwaway287391 wrote: post #30

billconan wrote: post #6

For the past week I have been frustrated by an opensource code of a deep learning paper. This type of things are so common in academia. The particular code I looked at has missing documentation, hardcoded local paths, broken dataset download links and broken pretrained model download links. I have to fix bugs before the code can run. I'm very curious how did the author run that code with the bugs. I call them insince…

This sort of thing is aggravating to read. Frankly it comes off as really entitled. As researchers, the expectation is now that we not only have to do the research and write a paper like the good old days, but we have to release the code too. Okay, fine. But now that's not enough either -- the code has to be well-documented and clean. Ugh, alright, fine -- it's going to take me a few extra weeks of not doing research…

If it was just one thing or the other, perhaps he wouldn't have been so critical of it. People are going to understand if you require a few paths to be changed or some files to be downloaded manually into specific relative locations. The ask is only that somebody should be able to use the code without doing their own novel research, and be able to verify it's doing what it is claimed to be doing. If this means your download links need to point to the right location, the code has to run in a location other than /home/hellokitty/ml, and that it needs to meet standards of readability—well, good, that's what the scientific process is about.

Re: Differences between the word2vec paper and its implementation

#43

Post by throwaway287391 » Mon, Jun 03, 2019, 11:50 PM UTC

liquidify wrote: post #38

throwaway287391 wrote: post #30

Earlier quoted context omitted.

This sort of thing is aggravating to read. Frankly it comes off as really entitled. As researchers, the expectation is now that we not only have to do the research and write a paper like the good old days, but we have to release the code too. Okay, fine. But now that's not enough either -- the code has to be well-documented and clean. Ugh, alright, fine -- it's going to take me a few extra weeks of not doing research…

If there are bugs in your code that would have prevented you from being able to do what you claim to do, then to the readers of your paper, it seems more likely that you might not have performed the experimentation that you claim to.

What? This has nothing to do with bugs in the code that would've affected the original author's ability to run the original experiments described in the paper. The user I replied to is talking about things like hard-coded data paths and wanting to be able to run the code at the press of a button on any platform. The expectation for this level of cross-platform compatibility is difficult to achieve for engineering firms with millions to invest in making it work properly. He's mad that individual machine learning researchers don't always do a great job of making it happen for their cutting-edge research which was a difficult enough engineering task to get running on their machine.

Re: Differences between the word2vec paper and its implementation

#44

Post by gubbrora » Mon, Jun 03, 2019, 11:54 PM UTC

bollu wrote: post #5

newen wrote: post #2

This kind of things happens all the time in academia. The authors are either constrained by space due to paper limitations or they are too lazy to explain all the little details that go into the algorithm. I used to do research in computer vision a few years ago and it used to be that people won't publish their code and they purposely won't put in all of the details of the algorithm in the paper. Many of those algori…

Is this really that common? That's disheartening, I want to spend time in academia but experiences like this are sucking the fun out for me...

Yes it's common that a lot of "little details" are left out. Yes it does make things harder. No it's not a big enough problem that you need to be disheartened. You could even see it as an opportunity to raise the bar and stand out positively.

Re: Differences between the word2vec paper and its implementation

#45

Post by randcraw » Mon, Jun 03, 2019, 11:57 PM UTC

liquidify wrote: post #38

throwaway287391 wrote: post #30

Earlier quoted context omitted.

This sort of thing is aggravating to read. Frankly it comes off as really entitled. As researchers, the expectation is now that we not only have to do the research and write a paper like the good old days, but we have to release the code too. Okay, fine. But now that's not enough either -- the code has to be well-documented and clean. Ugh, alright, fine -- it's going to take me a few extra weeks of not doing research…

If there are bugs in your code that would have prevented you from being able to do what you claim to do, then to the readers of your paper, it seems more likely that you might not have performed the experimentation that you claim to.

Agreed. If your results are not reproducible by others, the publication should issue a retraction and withdraw your work. Code that works is essential.

Re: Differences between the word2vec paper and its implementation

#46

Post by caenorst » Mon, Jun 03, 2019, 11:58 PM UTC

throwaway287391 wrote: post #30

billconan wrote: post #6

For the past week I have been frustrated by an opensource code of a deep learning paper. This type of things are so common in academia. The particular code I looked at has missing documentation, hardcoded local paths, broken dataset download links and broken pretrained model download links. I have to fix bugs before the code can run. I'm very curious how did the author run that code with the bugs. I call them insince…

This sort of thing is aggravating to read. Frankly it comes off as really entitled. As researchers, the expectation is now that we not only have to do the research and write a paper like the good old days, but we have to release the code too. Okay, fine. But now that's not enough either -- the code has to be well-documented and clean. Ugh, alright, fine -- it's going to take me a few extra weeks of not doing research…

I have spent enough time to debunk papers to know that a lot of researchers are shamelessly cheating or make mistakes (as EVERYBODY does). Reviewing your code should be part of the peer review process.

Sorry but I don't trust your research just based on yours words and cherry picked images.

Also research is incremental, so producing a proper code on which other can work on top of should be part of the CONTRIBUTION.

Re: Differences between the word2vec paper and its implementation

#47

Post by gcommer » Mon, Jun 03, 2019, 11:59 PM UTC

throwaway287391 wrote: post #43

liquidify wrote: post #38

Earlier quoted context omitted.

If there are bugs in your code that would have prevented you from being able to do what you claim to do, then to the readers of your paper, it seems more likely that you might not have performed the experimentation that you claim to.

What? This has nothing to do with bugs in the code that would've affected the original author's ability to run the original experiments described in the paper. The user I replied to is talking about things like hard-coded data paths and wanting to be able to run the code at the press of a button on any platform. The expectation for this level of cross-platform compatibility is difficult to achieve for engineering fir…

Reread the original post:

> I have to fix bugs before the code can run. I'm very curious how did the author run that code with the bugs.

Re: Differences between the word2vec paper and its implementation

#48

Post by namibj » Tue, Jun 04, 2019, 12:14 AM UTC

throwaway287391 wrote: post #30

billconan wrote: post #6

For the past week I have been frustrated by an opensource code of a deep learning paper. This type of things are so common in academia. The particular code I looked at has missing documentation, hardcoded local paths, broken dataset download links and broken pretrained model download links. I have to fix bugs before the code can run. I'm very curious how did the author run that code with the bugs. I call them insince…

This sort of thing is aggravating to read. Frankly it comes off as really entitled. As researchers, the expectation is now that we not only have to do the research and write a paper like the good old days, but we have to release the code too. Okay, fine. But now that's not enough either -- the code has to be well-documented and clean. Ugh, alright, fine -- it's going to take me a few extra weeks of not doing research…

I'd much prefer some code with a license that doesn't make looking at it a hazard for writing (A)GPL later, than to have to read the paper many times until I figure out how they meant to combine loads of prior research that is only mentioned to exist on the side, without any info on whether it was used in this or not.

Being able to compile in some way and, possibly after some bug-fixing/compiler-pleasing yielding useful results that approach/relate to those in the paper, greatly increases my confidence in the researcher and his work. Pretty code is nice, but it's weighed separately, not affecting the appreciation for the original research work itself. Only the existence of code with an AGPL or less restricted license affects how I perceive the paper/original research.

Re: Differences between the word2vec paper and its implementation

#49

Post by throwaway287391 » Tue, Jun 04, 2019, 12:17 AM UTC

gcommer wrote: post #47

throwaway287391 wrote: post #43

Earlier quoted context omitted.

What? This has nothing to do with bugs in the code that would've affected the original author's ability to run the original experiments described in the paper. The user I replied to is talking about things like hard-coded data paths and wanting to be able to run the code at the press of a button on any platform. The expectation for this level of cross-platform compatibility is difficult to achieve for engineering fir…

Reread the original post: > I have to fix bugs before the code can run. I'm very curious how did the author run that code with the bugs.

You're right, I misread that part.

Re: Differences between the word2vec paper and its implementation

#50

Post by newen » Tue, Jun 04, 2019, 12:52 AM UTC

chriskanan wrote: post #21

newen wrote: post #2

This kind of things happens all the time in academia. The authors are either constrained by space due to paper limitations or they are too lazy to explain all the little details that go into the algorithm. I used to do research in computer vision a few years ago and it used to be that people won't publish their code and they purposely won't put in all of the details of the algorithm in the paper. Many of those algori…

Completely agree. The situation today is far better than a decade ago, with code releases for machine learning and computer vision papers being much more common than before. I try to make my students release polished and easy to use code, but it can sometimes fall through the cracks due to deadlines, etc. Many projects are the output of a single PhD student.

Yep, it's so much better now in computer vision with early publishing in arxiv and published code. I feel like that is one of the reasons why research in CV is progressing so fast.

Also, I think it is something of a fact of life that you can't put all of the details of your algorithm in the paper, not just paper length limitations. I have actually tried to do this in two papers by putting all of the details in the supplements and the work to explain all the details and justify my choice of parameters and decisions for edge cases is almost as hard as writing the main paper. It becomes hard to justify the time spent pretty fast. Also, putting this in the main paper makes your beautiful explanations be tinged with edge cases and digressions haha.