Live data from Hacker News

Differences between the word2vec paper and its implementation

github.com

11–20 of 156 posts

Re: Differences between the word2vec paper and its implementation

#11

For the past week I have been frustrated by an opensource code of a deep learning paper. This type of things are so common in academia. The particular code I looked at has missing documentation, hardcoded local paths, broken dataset download links and broken pretrained model download links. I have to fix bugs before the code can run. I'm very curious how did the author run that code with the bugs. I call them insince…

Have you been able to reach out to the author and get clarification?

I tried to file bugs on github, only to find out that people (sometimes 2 years ahead me) was blocked by the same issue and there are no answers.

They didn't even bother to merge my pull requests.

Re: Differences between the word2vec paper and its implementation

#12

For the past week I have been frustrated by an opensource code of a deep learning paper. This type of things are so common in academia. The particular code I looked at has missing documentation, hardcoded local paths, broken dataset download links and broken pretrained model download links. I have to fix bugs before the code can run. I'm very curious how did the author run that code with the bugs. I call them insince…

It is very common. Scientist code isn't software engineering code (and software engineering code is very often worthy of criticism)

It's just good news when you can find source code at all instead of just being told vague things about something the author did.

Re: Differences between the word2vec paper and its implementation

#13

For the past week I have been frustrated by an opensource code of a deep learning paper. This type of things are so common in academia. The particular code I looked at has missing documentation, hardcoded local paths, broken dataset download links and broken pretrained model download links. I have to fix bugs before the code can run. I'm very curious how did the author run that code with the bugs. I call them insince…

Maintaining a pipeline that works takes effort that can be spent somewhere else. As long as it can be reproduced with a little effort from me, I really don't mind. I would rather they spend time to try a lot of different things, build intuitions, and search better backbone structures than maintaining a CI system so that every time their code updates can train imageNet to That should be left to TensorFlow model zoo or GluonCV model zoo. I simply look at these research "open-source" as reproducible research.

Re: Differences between the word2vec paper and its implementation

#14
post #13

For the past week I have been frustrated by an opensource code of a deep learning paper. This type of things are so common in academia. The particular code I looked at has missing documentation, hardcoded local paths, broken dataset download links and broken pretrained model download links. I have to fix bugs before the code can run. I'm very curious how did the author run that code with the bugs. I call them insince…

Maintaining a pipeline that works takes effort that can be spent somewhere else. As long as it can be reproduced with a little effort from me, I really don't mind. I would rather they spend time to try a lot of different things, build intuitions, and search better backbone structures than maintaining a CI system so that every time their code updates can train imageNet to That should be left to TensorFlow model zoo or…

Yeah but I wonder what fraction of CS papers test the algorithm that they thought they were testing...

Re: Differences between the word2vec paper and its implementation

#15
post #5
post #2

This kind of things happens all the time in academia. The authors are either constrained by space due to paper limitations or they are too lazy to explain all the little details that go into the algorithm. I used to do research in computer vision a few years ago and it used to be that people won't publish their code and they purposely won't put in all of the details of the algorithm in the paper. Many of those algori…

Is this really that common? That's disheartening, I want to spend time in academia but experiences like this are sucking the fun out for me...

Yes, this was a huge disappointment when I read Chemistry & Physics years ago. The naive view of papers was that they were to move human knowledge forward, but it became clear they were an elaborate knowledge withholding device!

Fortunately the trend seems to be towards better levels of openness, but it varies by subject. Stories like the BASF team being unable to reproduce vast numbers of established published techniques are way too common.

Re: Differences between the word2vec paper and its implementation

#16
post #5
post #2

This kind of things happens all the time in academia. The authors are either constrained by space due to paper limitations or they are too lazy to explain all the little details that go into the algorithm. I used to do research in computer vision a few years ago and it used to be that people won't publish their code and they purposely won't put in all of the details of the algorithm in the paper. Many of those algori…

Is this really that common? That's disheartening, I want to spend time in academia but experiences like this are sucking the fun out for me...

Academia isn't flawless, it's no paradise. It's just humans and their factions, with all the good and bad that brings.

> The authors are ... constrained by space due to paper limitations

This is very real. Different journals have different criteria: word count; formatting; the number of tables you're permitted to include; etc. It's archaic and daft but that's the truth of it. And that's before we even get started on the really bonkers stuff like author order, impact factor, reviewer workload vs lack of pay, publish or perish, and so on.

Re: Differences between the word2vec paper and its implementation

#17
Speaking as someone who has read about 40 years of papers in compiler optimization, it's very interesting.

In the early days, there were fairly exact algorithms that worked as described, and were implemented as described, but were pseudocoded in papers. Where the pseudocode differed from implementation, differences were described in great detail (IE they may say an array can be shared but isn't to make the pseudocode easier to describe).

Then, over time, things start to get further away from that. You start to see papers published with algorithms that either don't work as described, or are so inefficient as to be unusable. Like literally cannot work. Where you can get source code, later looking at the source code shows that's not what they did at all.

One infamous example of this is SSAPRE - to this day, people have a lot of trouble understanding the paper (and it has significant errors that make the algorithm incorrect as written). The concept sure, but the exact algorithm - less so. Reading the source code to it in Open64 - it is just wildly different than the paper (and often requires a lot of thought for people to convince themselves it is correct).

It's not just better engineering/datastructures vs research algorithms.

The one shining counterexample is the Rice folks who wrote their massively scalar compiler in nuweb (one of many literate programming environments), so the descriptions/papers and code were in the same place - these are very very readable and useful papers in my experience.

Nowadays it's coming back to the earlier daysdue to github/et al. People seem to try to make the code more like the paper algorithm since they now release the code.

Word2vec appears to be a counterexample (maybe because they released the code they didn't feel a need to get the paper as right)

Re: Differences between the word2vec paper and its implementation

#18

For the past week I have been frustrated by an opensource code of a deep learning paper. This type of things are so common in academia. The particular code I looked at has missing documentation, hardcoded local paths, broken dataset download links and broken pretrained model download links. I have to fix bugs before the code can run. I'm very curious how did the author run that code with the bugs. I call them insince…

One explained to me here that the problem is they're mainly rewarded by funding agencies for how many papers they publish. That's why quality went down in general. Then, since funding agencies aren't rewarding code, they're either not encouraged to get it right or actively discouraged from doing it to focus on next paper.

This sad situation is both bad for open source and science. Bad for science since inaccurate results that probably aren't even reproducible aren't the knowledge advancements we need. Funding agencies like NSF need to change their policies to address the root cause here. Universities will follow where the incentives lead them.

Re: Differences between the word2vec paper and its implementation

#19
On a similar note, a long time ago I read the Doc2Vec paper, then looked at popular Doc2Vec implementations. They didn’t seem to do the same thing. The paper said you basically make vectors for words, then append on an additional space that represents the additional information of documents as opposed to single words.

All popular implementations I found seemed to put the document vectors into the same space as the word vectors. They also didn’t seem to do any better than a tf-idf weighted average of word vectors... curious if anyone has ever bumped against this.

Re: Differences between the word2vec paper and its implementation

#20
post #5
post #2

This kind of things happens all the time in academia. The authors are either constrained by space due to paper limitations or they are too lazy to explain all the little details that go into the algorithm. I used to do research in computer vision a few years ago and it used to be that people won't publish their code and they purposely won't put in all of the details of the algorithm in the paper. Many of those algori…

Is this really that common? That's disheartening, I want to spend time in academia but experiences like this are sucking the fun out for me...

[deleted]
Post reply on HN