Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Differences between the word2vec paper and its implementation

github.com

Post reply on HN
Print view View original

51–60 of 156 posts

Re: Differences between the word2vec paper and its implementation

#51

Post by KirinDave »

throwaway287391 wrote: post #30
billconan wrote: post #6

For the past week I have been frustrated by an opensource code of a deep learning paper. This type of things are so common in academia. The particular code I looked at has missing documentation, hardcoded local paths, broken dataset download links and broken pretrained model download links. I have to fix bugs before the code can run. I'm very curious how did the author run that code with the bugs. I call them insince…

This sort of thing is aggravating to read. Frankly it comes off as really entitled. As researchers, the expectation is now that we not only have to do the research and write a paper like the good old days, but we have to release the code too. Okay, fine. But now that's not enough either -- the code has to be well-documented and clean. Ugh, alright, fine -- it's going to take me a few extra weeks of not doing research…

Pardon me, maybe I just misunderstood the whole idea of research but what good is it if it's not reproducible?

I can understand it may be part of a meaningful personal journey for you, and I appreciate that. But if no one else can validate your research they're correct to discredit it and you.

So what is the optimal outcome here? Should we hold you to a standard of reproducibility even if it is as minimal as, "actually describe your algorithms correctly and don't misrepresent a piece of code and a paper?" Or should everyone just decide you can find your own research funding if it's not going to help anyone?

Re: Differences between the word2vec paper and its implementation

#52

Post by p1esk »

KirinDave wrote: post #51
throwaway287391 wrote: post #30

Earlier quoted context omitted.

This sort of thing is aggravating to read. Frankly it comes off as really entitled. As researchers, the expectation is now that we not only have to do the research and write a paper like the good old days, but we have to release the code too. Okay, fine. But now that's not enough either -- the code has to be well-documented and clean. Ugh, alright, fine -- it's going to take me a few extra weeks of not doing research…

Pardon me, maybe I just misunderstood the whole idea of research but what good is it if it's not reproducible? I can understand it may be part of a meaningful personal journey for you, and I appreciate that. But if no one else can validate your research they're correct to discredit it and you. So what is the optimal outcome here? Should we hold you to a standard of reproducibility even if it is as minimal as, "actual…

Pardon me, maybe I just misunderstood the whole idea of "reproducible" but it is only not reproducible if you tried to implement the ideas described in the paper yourself, and I mean really tried, and contacted the authors for help, and still failed to achieve the claimed results.

Re: Differences between the word2vec paper and its implementation

#53

Post by jfoutz »

throwaway287391 wrote: post #49
gcommer wrote: post #47

Earlier quoted context omitted.

Reread the original post: > I have to fix bugs before the code can run. I'm very curious how did the author run that code with the bugs.

You're right, I misread that part.

throwaway287391 it's hilarious how much you value your own opinion. You feel it's important your interlocutor knows you know you were corrected, while remaining anonymous.

This is like, the funniest thing i've seen all week.

You should run for office.

_edit_

Mmmm delicious downvotes. How many alts do you have throwaway287391? i don't think dang would or should expose you.

But it is even more funny to me that you would log in with an alt (2?!), specifically to downvote.

Anyway.I'm sure with some practice you'll get better at trolling little buddy.

Re: Differences between the word2vec paper and its implementation

#54

Post by seanmcdirmid »

KirinDave wrote: post #51
throwaway287391 wrote: post #30

Earlier quoted context omitted.

This sort of thing is aggravating to read. Frankly it comes off as really entitled. As researchers, the expectation is now that we not only have to do the research and write a paper like the good old days, but we have to release the code too. Okay, fine. But now that's not enough either -- the code has to be well-documented and clean. Ugh, alright, fine -- it's going to take me a few extra weeks of not doing research…

Pardon me, maybe I just misunderstood the whole idea of research but what good is it if it's not reproducible? I can understand it may be part of a meaningful personal journey for you, and I appreciate that. But if no one else can validate your research they're correct to discredit it and you. So what is the optimal outcome here? Should we hold you to a standard of reproducibility even if it is as minimal as, "actual…

If I have to go dig up someone’s code to reproduce their result because I don’t believe/understand the idea in its published form, they’ve already lost me.

Word2vec is a great example of a piece of research that conveys a great idea with plausible results, where in fact, the numbers are less useful than the story behind them. Of course, google is a big corp that can just put out useful interesting stuff without worrying about how to play the science game.

Re: Differences between the word2vec paper and its implementation

#55

Post by gojomo »

b_tterc_p wrote: post #19

On a similar note, a long time ago I read the Doc2Vec paper, then looked at popular Doc2Vec implementations. They didn’t seem to do the same thing. The paper said you basically make vectors for words, then append on an additional space that represents the additional information of documents as opposed to single words. All popular implementations I found seemed to put the document vectors into the same space as the wo…

The only code released by the 'Paragraph Vector' paper authors was a small patch, from Mikolov, that added paragraph-vectors to the original `word2vec.c` implementation in a very simple way: treating the 1st token of each line as a special paragraph-vector, still string-named (and allocated in the same lookup dictionary). Only by convention (a special prefix on those paragraph-vector tokens) could collisions with similarly-named word-vectors avoided.

That's a nice minimal way to demo/test the idea, but limited and fragile in other ways. The initial gensim implementation did something similar, then I changed it to use a separate doc-vectors space, to better support a lot of options (including the PV-DM mode with a concatenative input layer – which has never been confirmed to perform as well as the original paper implied).

Re: Differences between the word2vec paper and its implementation

#56

Post by seanmcdirmid »

PaulHoule wrote: post #14
liuliu wrote: post #13

Earlier quoted context omitted.

Maintaining a pipeline that works takes effort that can be spent somewhere else. As long as it can be reproduced with a little effort from me, I really don't mind. I would rather they spend time to try a lot of different things, build intuitions, and search better backbone structures than maintaining a CI system so that every time their code updates can train imageNet to That should be left to TensorFlow model zoo or…

Yeah but I wonder what fraction of CS papers test the algorithm that they thought they were testing...

A lot of CS papers aren’t testing algorithms at all.

Re: Differences between the word2vec paper and its implementation

#57

Post by archgoon »

KirinDave wrote: post #51
throwaway287391 wrote: post #30

Earlier quoted context omitted.

This sort of thing is aggravating to read. Frankly it comes off as really entitled. As researchers, the expectation is now that we not only have to do the research and write a paper like the good old days, but we have to release the code too. Okay, fine. But now that's not enough either -- the code has to be well-documented and clean. Ugh, alright, fine -- it's going to take me a few extra weeks of not doing research…

Pardon me, maybe I just misunderstood the whole idea of research but what good is it if it's not reproducible? I can understand it may be part of a meaningful personal journey for you, and I appreciate that. But if no one else can validate your research they're correct to discredit it and you. So what is the optimal outcome here? Should we hold you to a standard of reproducibility even if it is as minimal as, "actual…

The original idea behind "reproducible" is that the ideas conveyed in the paper should be enough to reproduce the results. Physicists and biologists are not expected to drive over to your lab to figure out what's wrong with your setup.

Now, that said, reproducibility is terrible in many fields. CS has an opportunity to act as a trailblazer here, but it should be noted that this would be holding themselves to a higher standard than their peers in other fields. As a result, there's going to be a learning process for everyone as they figure out how to make this all work. :)

Re: Differences between the word2vec paper and its implementation

#58

Post by p1esk »

liquidify wrote: post #38
throwaway287391 wrote: post #30

Earlier quoted context omitted.

This sort of thing is aggravating to read. Frankly it comes off as really entitled. As researchers, the expectation is now that we not only have to do the research and write a paper like the good old days, but we have to release the code too. Okay, fine. But now that's not enough either -- the code has to be well-documented and clean. Ugh, alright, fine -- it's going to take me a few extra weeks of not doing research…

If there are bugs in your code that would have prevented you from being able to do what you claim to do, then to the readers of your paper, it seems more likely that you might not have performed the experimentation that you claim to.

Typically when I post code for my papers, I try to clean it up, because it's an ugly mess that is hard to understand (even myself sometimes). This clean up process might introduce bugs. I'm not going to thoroughly test everything, I have next paper deadline coming up and my adviser could not care less about my code. But, if you're interested in my research and having trouble reproducing my results, email me and I'll help you.

Re: Differences between the word2vec paper and its implementation

#59

Post by ur-whale »

nullwasamistake wrote: post #39
DannyBee wrote: post #17

Speaking as someone who has read about 40 years of papers in compiler optimization, it's very interesting. In the early days, there were fairly exact algorithms that worked as described, and were implemented as described, but were pseudocoded in papers. Where the pseudocode differed from implementation, differences were described in great detail (IE they may say an array can be shared but isn't to make the pseudocode…

I ran into the same thing when implementing a hash table sorting algorithm from a recent paper. Nowhere in the paper does it mention that table size MUST be a power of two. The new table structure is supposedly more memory efficient than anything else out there. That's the whole point of it. So it's nice that they leave out "btw only works with size power of two". I got 2/3 of the way through an optimized implementat…

I think it's ok to names names in this specific instance: you'll save other people a lot of pain.

Re: Differences between the word2vec paper and its implementation

#60

Post by ur-whale »

throwaway287391 wrote: post #30
billconan wrote: post #6

For the past week I have been frustrated by an opensource code of a deep learning paper. This type of things are so common in academia. The particular code I looked at has missing documentation, hardcoded local paths, broken dataset download links and broken pretrained model download links. I have to fix bugs before the code can run. I'm very curious how did the author run that code with the bugs. I call them insince…

This sort of thing is aggravating to read. Frankly it comes off as really entitled. As researchers, the expectation is now that we not only have to do the research and write a paper like the good old days, but we have to release the code too. Okay, fine. But now that's not enough either -- the code has to be well-documented and clean. Ugh, alright, fine -- it's going to take me a few extra weeks of not doing research…

You might want to read up on docker.
Post reply on HN