Differences between the word2vec paper and its implementation
1–10 of 156 posts
Re: Differences between the word2vec paper and its implementation
#2This kind of things happens all the time in academia. The authors are either constrained by space due to paper limitations or they are too lazy to explain all the little details that go into the algorithm.
I used to do research in computer vision a few years ago and it used to be that people won't publish their code and they purposely won't put in all of the details of the algorithm in the paper. Many of those algorithms were patent pending and I assume the authors were hoping to make some money from the patents. Compared to that, it's a lot better nowadays where most of the popular papers come with published code.
Re: Differences between the word2vec paper and its implementation
#3This kind of things happens all the time in academia. The authors are either constrained by space due to paper limitations or they are too lazy to explain all the little details that go into the algorithm. I used to do research in computer vision a few years ago and it used to be that people won't publish their code and they purposely won't put in all of the details of the algorithm in the paper. Many of those algori…
[deleted]
Re: Differences between the word2vec paper and its implementation
#4My intuition for that, and you can tell me if its wrong.
The normal explanation for Word2Vec is 2 weight matrices, so the formula looks like this: (One_hot_input x W1) x W2, which is then softmaxed.
W1 then is the matrix that contain our focus embedding from, but if we only evaluate specific words on the target side, then W2 are actually our context embeddings, and the normal multiplication then is focus_w x context_w.
Am I wrong?
Re: Differences between the word2vec paper and its implementation
#5This kind of things happens all the time in academia. The authors are either constrained by space due to paper limitations or they are too lazy to explain all the little details that go into the algorithm. I used to do research in computer vision a few years ago and it used to be that people won't publish their code and they purposely won't put in all of the details of the algorithm in the paper. Many of those algori…
Is this really that common? That's disheartening, I want to spend time in academia but experiences like this are sucking the fun out for me...
Re: Differences between the word2vec paper and its implementation
#6For the past week I have been frustrated by an opensource code of a deep learning paper. This type of things are so common in academia. The particular code I looked at has missing documentation, hardcoded local paths, broken dataset download links and broken pretrained model download links. I have to fix bugs before the code can run. I'm very curious how did the author run that code with the bugs.
I call them insincerely opensourced projects.
Re: Differences between the word2vec paper and its implementation
#7My intuition for that, and you can tell me if its wrong. The normal explanation for Word2Vec is 2 weight matrices, so the formula looks like this: (One_hot_input x W1) x W2, which is then softmaxed. W1 then is the matrix that contain our focus embedding from, but if we only evaluate specific words on the target side, then W2 are actually our context embeddings, and the normal multiplication then is focus_w x context_…
Now it's `one_hot_focus x W1 x (one_hot_context x W2)^T`. So we still pick one row of the matrix from the focus and context embeddings, but they're separate embeddings.
Re: Differences between the word2vec paper and its implementation
#8For the past week I have been frustrated by an opensource code of a deep learning paper. This type of things are so common in academia. The particular code I looked at has missing documentation, hardcoded local paths, broken dataset download links and broken pretrained model download links. I have to fix bugs before the code can run. I'm very curious how did the author run that code with the bugs. I call them insince…
Have you been able to reach out to the author and get clarification?
Re: Differences between the word2vec paper and its implementation
#9This kind of things happens all the time in academia. The authors are either constrained by space due to paper limitations or they are too lazy to explain all the little details that go into the algorithm. I used to do research in computer vision a few years ago and it used to be that people won't publish their code and they purposely won't put in all of the details of the algorithm in the paper. Many of those algori…
Is this really that common? That's disheartening, I want to spend time in academia but experiences like this are sucking the fun out for me...
welcome to earth. we apologize for the mess.
Re: Differences between the word2vec paper and its implementation
#10My intuition for that, and you can tell me if its wrong. The normal explanation for Word2Vec is 2 weight matrices, so the formula looks like this: (One_hot_input x W1) x W2, which is then softmaxed. W1 then is the matrix that contain our focus embedding from, but if we only evaluate specific words on the target side, then W2 are actually our context embeddings, and the normal multiplication then is focus_w x context_…
Now it's `one_hot_focus x W1 x (one_hot_context x W2)^T`. So we still pick one row of the matrix from the focus and context embeddings, but they're separate embeddings.
Yes, but thats also what happens in the normal formulation, no? So the second weight matrix actually are our context embeddings?