Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Differences between the word2vec paper and its implementation

github.com

Post reply on HN
Print view View original

31–40 of 156 posts

Re: Differences between the word2vec paper and its implementation

#31

Post by exgrv »

bollu wrote: post #29
utopcell wrote: post #27

I think it very unfair to the original set of word2vec papers to be talking about 'academic dishonesty'. This is a case of a user that has little to no experience with neural networks. There are a ton of articles describing the need for random initialization [1][2]. In fact, if one spends a few seconds thinking about it, the need is evident. Without it, the NN cannot perform symmetry breaking: If inputs are set to ze…

That's hardly the point of the article --- the actual paper does not describe the use of two separate vectors for each word. The initialization was an interesting tidbit.

Except it does? After Equation 2: "v_w and v'_w are the input and output vector representations of w."

Re: Differences between the word2vec paper and its implementation

#32

Post by avn2109 »

bollu wrote: post #7
MichaelStaniek wrote: post #4

My intuition for that, and you can tell me if its wrong. The normal explanation for Word2Vec is 2 weight matrices, so the formula looks like this: (One_hot_input x W1) x W2, which is then softmaxed. W1 then is the matrix that contain our focus embedding from, but if we only evaluate specific words on the target side, then W2 are actually our context embeddings, and the normal multiplication then is focus_w x context_…

Now it's `one_hot_focus x W1 x (one_hot_context x W2)^T`. So we still pick one row of the matrix from the focus and context embeddings, but they're separate embeddings.

New user "MichaelStaniek" has a grammatical, relevant, good-faith sibling comment to this which is (imho) inexplicably banned, almost certainly due to a mistake, and I hope somebody will unban him.

Re: Differences between the word2vec paper and its implementation

#34

Post by RyEgswuCsn »

I find the title of the article rather exaggerating...

As of the first difference pointed out in the article, one of the CS224D lectures on word2vec did addressed it:

https://youtu.be/aRqn8t1hLxs?t=2650

It was also mentioned later in the lecture that having two vectors representing each word is meant to make the optimisation easier (so it's kind of a trick); at the end, the two vectors learnt will have to be averaged over in order to reach a single vector for each word.

To be fair, the fact that each word is represented by two vectors was also mentioned in the original paper describing word2vec:

https://arxiv.org/pdf/1310.4546.pdf

On page 3, just beneath equation (2).

Why so surprised?

Re: Differences between the word2vec paper and its implementation

#35

Post by YeGoblynQueenne »

not2b wrote: post #28
nickpsecurity wrote: post #18

Earlier quoted context omitted.

One explained to me here that the problem is they're mainly rewarded by funding agencies for how many papers they publish. That's why quality went down in general. Then, since funding agencies aren't rewarding code, they're either not encouraged to get it right or actively discouraged from doing it to focus on next paper. This sad situation is both bad for open source and science. Bad for science since inaccurate res…

Then some other academic can get credit for publishing a rigorous paper describing the actual algorithm, explaining why the original paper is wrong (without speculating on the motive, just the facts), and why the approach taken in the code is in fact better. Bonus points for describing a further improvements, with working code.

In machine learning, you don't get credit for publishing rigorous papers. You get credits for publishing papers that show improved performance:

One big challenge the community faces is that if you want to get a paper published in machine learning now it's got to have a table in it, with all these different data sets across the top, and all these different methods along the side, and your method has to look like the best one. If it doesn’t look like that, it’s hard to get published. I don't think that's encouraging people to think about radically new ideas.

Now if you send in a paper that has a radically new idea, there's no chance in hell it will get accepted, because it's going to get some junior reviewer who doesn't understand it. Or it’s going to get a senior reviewer who's trying to review too many papers and doesn't understand it first time round and assumes it must be nonsense. Anything that makes the brain hurt is not going to get accepted. And I think that's really bad.

Geoff Hinton interview:

https://www.wired.com/story/googles-ai-guru-computers-think-...

I guess people look at statistical machine learning and deep learning, see all the formulae and hear all the calculus terminology and think - "oh, wow, that's a really rigorous field! Look at all the formalisms!".

But it's not. It's an extremely, almost exclusively, empirical field. The mathiness and the formulae are just unfortunate attempts to pass off the whole endeavour as something that it's not- some kind of careful science that uncovers deep truths about intelligence and cognition. In truth, it's all just about beating other peoples' systems in very specific benchmarks.

If it wasn't for this culture of pretensions to higher science, machine learning papers would most likely be written with much more clarity than they are now and mistakes like the one described in the above article would be rare.

Re: Differences between the word2vec paper and its implementation

#36

Post by xxxpupugo »

The title reads to me like hyperbole.

The implementation can differ, they got time to refactor/optimize it after the publication. But they can't probably revise the paper itself. As long as the code is there and can produce said/better result, then it is probably your responsibility to keep the differences in check.

It is actually quite common for deep learning papers overall, the github repo gets updated after the paper is out, and you will find the divergence lying there.

Re: Differences between the word2vec paper and its implementation

#37

Post by eggie5 »

instead of thinking about what it is in practice: skip-gram negative sampling, I think it's much more intuitive to think about what it is in theory: extreme multi-class classification.

word2vec is a multi-class classification problem with a softmax output layer and cross-entropy loss. The novel part of word2vec, in my opinion, is two:

1. dataset (proximal input word & output word) generation from documents eg: skiagram, CBOW, etc 2. engineering speedup for softmax: Approximate Softmax eg Negative Sampling using NCE, hierarchal softmax, etc

If you just build word2vec w/o step 2, it's a easier to understand. Then when you get that working, add in the negative sampling speedup trick which isn't core the theoretical algorithm.

Re: Differences between the word2vec paper and its implementation

#38

Post by liquidify »

throwaway287391 wrote: post #30
billconan wrote: post #6

For the past week I have been frustrated by an opensource code of a deep learning paper. This type of things are so common in academia. The particular code I looked at has missing documentation, hardcoded local paths, broken dataset download links and broken pretrained model download links. I have to fix bugs before the code can run. I'm very curious how did the author run that code with the bugs. I call them insince…

This sort of thing is aggravating to read. Frankly it comes off as really entitled. As researchers, the expectation is now that we not only have to do the research and write a paper like the good old days, but we have to release the code too. Okay, fine. But now that's not enough either -- the code has to be well-documented and clean. Ugh, alright, fine -- it's going to take me a few extra weeks of not doing research…

If there are bugs in your code that would have prevented you from being able to do what you claim to do, then to the readers of your paper, it seems more likely that you might not have performed the experimentation that you claim to.

Re: Differences between the word2vec paper and its implementation

#39

Post by nullwasamistake »

DannyBee wrote: post #17

Speaking as someone who has read about 40 years of papers in compiler optimization, it's very interesting. In the early days, there were fairly exact algorithms that worked as described, and were implemented as described, but were pseudocoded in papers. Where the pseudocode differed from implementation, differences were described in great detail (IE they may say an array can be shared but isn't to make the pseudocode…

I ran into the same thing when implementing a hash table sorting algorithm from a recent paper. Nowhere in the paper does it mention that table size MUST be a power of two.

The new table structure is supposedly more memory efficient than anything else out there. That's the whole point of it.

So it's nice that they leave out "btw only works with size power of two". I got 2/3 of the way through an optimized implementation before I realized it wasn't any more efficient than the previous contender, in fact much less so, if you couldn't fit your set into exactly a power of two size.

Not outright lying but creative ommission for sure

Post reply on HN