Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Differences between the word2vec paper and its implementation

github.com

Post reply on HN
Print view View original

131–140 of 156 posts

Re: Differences between the word2vec paper and its implementation

#131

Post by andreasvc »

bollu wrote: post #29
utopcell wrote: post #27

I think it very unfair to the original set of word2vec papers to be talking about 'academic dishonesty'. This is a case of a user that has little to no experience with neural networks. There are a ton of articles describing the need for random initialization [1][2]. In fact, if one spends a few seconds thinking about it, the need is evident. Without it, the NN cannot perform symmetry breaking: If inputs are set to ze…

That's hardly the point of the article --- the actual paper does not describe the use of two separate vectors for each word. The initialization was an interesting tidbit.

bollu, multiple people have shown that your claim that the paper doesn't match the code is flat out wrong. I think at this point you should issue a retraction of your wildly inappropriate suggestion of academic dishonesty.

Re: Differences between the word2vec paper and its implementation

#132

Post by kaitai »

Qworg wrote: post #92
p1esk wrote: post #90

Earlier quoted context omitted.

And who exactly is supposed to verify that code is reproducible for all the papers that get accepted?

The editors, previous to accepting. If it doesn't run out of the box and reproduce the result, it doesn't get published.

Agree with sibling commenters that editors shouldn't be doing this themselves -- in my field, for instance, an editor is just the paid-in-prestige prof who agrees to look through articles submitted in a certain area, find referees for the paper, cajole the referees to review it at 6 months, cajole the referees again at one year, etc., and then make a decision based on the referee reports. (Clearly I'm not in CS as a 1.5-year turnaround is not so nuts in my field.)

Anyhow, I think Qworg has a point in that the editors do have responsibility for what they accept. It's true that referees do not have the resources to reproduce results: I for instance have limited time on the shared high-performance computing resources we have, so I'm not going to use it on someone else's paper. However, as another commenter says, this could be paid for. A journal could potentially "sell" this as a competitive advantage: papers in this journal were actually reproduced, by paid staff who are neither editors nor referees, people paid in currency that can be exchanged for goods (not academic currency).

Re: Differences between the word2vec paper and its implementation

#133

Post by kaitai »

nullc wrote: post #82
billconan wrote: post #6

For the past week I have been frustrated by an opensource code of a deep learning paper. This type of things are so common in academia. The particular code I looked at has missing documentation, hardcoded local paths, broken dataset download links and broken pretrained model download links. I have to fix bugs before the code can run. I'm very curious how did the author run that code with the bugs. I call them insince…

There appears to be something of a software engineering crisis in academia. Imagine that you are a graduate student or a newly minted PHD who happens to be pretty good at practical software engineering: You can leave academia for FAANG and a solid six figure income, or you can stay struggling, poorly paid, unable to get tenure in academia. In some groups software engineers are treated as glorified typists, too valuab…

I just wish I could upvote your comment three more times.

To compound the problem, the fact that you're competing against many of your peers for the few jobs there are, and jumping positions every 1-3 years after grad school, means that best practices are not disseminated by working with a stable set of coworkers. This is similar to your 4th paragraph point.

As a mentor of grad students on projects that these days are increasingly using machine learning, I am trying to figure out how to add this stuff to my teaching load, essentially, because the pain of my students using nested for loops when they could just use a single vectorized expression is real... but I myself have no formal CS training......

Re: Differences between the word2vec paper and its implementation

#134

Post by kaitai »

patrick5415 wrote: post #110
not2b wrote: post #28

Earlier quoted context omitted.

Then some other academic can get credit for publishing a rigorous paper describing the actual algorithm, explaining why the original paper is wrong (without speculating on the motive, just the facts), and why the approach taken in the code is in fact better. Bonus points for describing a further improvements, with working code.

Yet there are approximately 0 papers doing that. Flawed or not not, the original paper still beats the socks off your approach on novelty. And (perceived) novelty is the lifeblood of the acedemic.

There is a phenomenon in academia I call the "pissing on a lamppost" phenomenon. A well-known researcher mentions his thoughts on (topic) in the penultimate paragraph of a paper on something else. When you talk to them, they say "oh, I don't actually know how to do that, but I think it should work!" So some sucker like me goes & figures out how it actually works, but... BigName has already discussed it so it's not a new result (see citation for penultimate paragraph) so we are not interested in publishing it :)

You can get around it if you show BigName is wrong but if they've been vague enough that's hard.

Re: Differences between the word2vec paper and its implementation

#135

Post by patrick5415 »

cfcf14 wrote: post #118
patrick5415 wrote: post #103

Earlier quoted context omitted.

Reproducible research does not mean you don’t have to do any work. The original author isn’t going to come to your lab and clean your pipettes nor should they. Grow up.

The research published in this case (the original word2vec paper) is quite literally not what it claims to be. It is factually incorrect. Is that really the responsibility of the individual attempting to reproduce the work to identify and correct? How wrong should the original work be before it ceases to be a case of 'clean your own pipettes'?

I’d say at the same point that others in the field would generally agree it’s fradulent. I’m not familiar enough with the field of word2vec to say if this is the case or not. But regardless, asking that the implementation be representative of the paper is a lot different than demanding docker containers and install scripts.

Re: Differences between the word2vec paper and its implementation

#136

Post by curl_e »

raverbashing wrote: post #98
colechristensen wrote: post #12

Earlier quoted context omitted.

It is very common. Scientist code isn't software engineering code (and software engineering code is very often worthy of criticism) It's just good news when you can find source code at all instead of just being told vague things about something the author did.

> Scientist code isn't software engineering code (and software engineering code is very often worthy of criticism) Absolutely correct, and not in a good way Though to be fair, people doing research have other interests besides maintainable code (though half or more of the annoyances will probably come and bite them later)

The good news is people are trying to address this. Starting in the UK there is a movement for Research Software Engineering https://rse.ac.uk/ and happening right now is the German RSE conference https://www.de-rse.org/en/conf2019/

Re: Differences between the word2vec paper and its implementation

#137

Post by p1esk »

elcomet wrote: post #111
sampo wrote: post #107

Earlier quoted context omitted.

> It's not the editor's job to replicate the results. They obviously meant the reviewers. Not everyone is intimately familiar with academic terminology.

The reviewers are commissioned by the editor, so I would say it's the same thing. Unless you meant the reviewers should replicate the papers with their own ressources, but I think it's not what we want. Reviewers are benevolent, and they don't have time nor resources to replicate results. Maybe a solution would be a platform, like a CI for machine learning where authors would send their codes, and the CI would run it…

Please keep in mind that currently there are a lot of unqualified people reviewing papers, even at top conferences, as evidenced by OpenReview.net. There was an experiment done at NIPS a few years ago where something like half of the rejected papers got accepted after having been reviewed independently by different reviewers, and/or the other way around. If people can't even agree if paper is good or not, while assuming all results are reproducible, then we have a bigger problem.

To me, getting the right people to review the paper is more important than reproducible code.

Re: Differences between the word2vec paper and its implementation

#138

Post by scott_s »

p1esk wrote: post #137
elcomet wrote: post #111

Earlier quoted context omitted.

The reviewers are commissioned by the editor, so I would say it's the same thing. Unless you meant the reviewers should replicate the papers with their own ressources, but I think it's not what we want. Reviewers are benevolent, and they don't have time nor resources to replicate results. Maybe a solution would be a platform, like a CI for machine learning where authors would send their codes, and the CI would run it…

Please keep in mind that currently there are a lot of unqualified people reviewing papers, even at top conferences, as evidenced by OpenReview.net. There was an experiment done at NIPS a few years ago where something like half of the rejected papers got accepted after having been reviewed independently by different reviewers, and/or the other way around. If people can't even agree if paper is good or not, while assum…

There is a difference between unqualified people, and qualified but busy people. While there may be unqualified people reviewing papers, even qualified people who don't put in enough time and effort to review a paper comes to the same result. But the solution for that problem will look different.

There's also a big difference between accepting a paper and judging if it is good. At a top conference like NIPS, lots of good papers will get rejected. That's the nature of being a top conference in a hot field. It's similar to a high school valedictorian not getting into Harvard: it's not a judgement that they are a bad student. But when you're an elite thing, and lots of people want in, you end up looking for reasons to reject rather than accept. You will end up rejecting good papers/students in such a case. NIPS, or NeurIPS now, is a particularly interesting example, as a colleague recently told me that they just got 12,000 submissions this year.

There's also an issue in computer science papers where people may not like your paper based on personal taste. They may not be convinced you're solving a real problem, and if your paper involves a new artifact, the design of it. I refer to these as "your baby is ugly" reviews. They don't think you're wrong, they just don't like it.

I think people's time is a bigger thing than the wrong people. One problem, I think, is that we tend to compete for these once-a-year conferences, causing an enormous time crunch for reviewers. VLDB has a great model where they do rolling acceptance throughout the year, accepting a few papers every month, even though the conference happens once a year (http://vldb.org/2019/?submission-guidelines). I believe this is the best hybrid between the typical CS conference system, and the more common journal system in the rest of science and engineering.

Re: Differences between the word2vec paper and its implementation

#139

Post by scott_s »

billconan wrote: post #11
GuyPostington wrote: post #8

Earlier quoted context omitted.

Have you been able to reach out to the author and get clarification?

I tried to file bugs on github, only to find out that people (sometimes 2 years ahead me) was blocked by the same issue and there are no answers. They didn't even bother to merge my pull requests.

There's probably a disconnect in expectations. The researchers are probably publishing the code more as a document, to say, hey, this is how we did this. It sounds like you have expectations that the code is part of an ongoing project. But ongoing projects require maintenance, and the researchers have likely moved on to new things.

Re: Differences between the word2vec paper and its implementation

#140

Post by rurban »

bjourne wrote: post #120
nullwasamistake wrote: post #39

Earlier quoted context omitted.

I ran into the same thing when implementing a hash table sorting algorithm from a recent paper. Nowhere in the paper does it mention that table size MUST be a power of two. The new table structure is supposedly more memory efficient than anything else out there. That's the whole point of it. So it's nice that they leave out "btw only works with size power of two". I got 2/3 of the way through an optimized implementat…

Backing arrays for fast hash tables are always powers of 2. That's how they are implemented in all mainstream languages. Details are often omitted in academia because of publishing constraints or because the authors thought they were obvious.

Fast yes, but more insecure. Using prime sized tables will get all hash bits being used. With power of 2 you can get away with the mod, but it's easier to crack, and the grow factor is bigger than the ideal fibonacci sequence, or a prime sequence.
Post reply on HN