Differences between the word2vec paper and its implementation • Hacker News Board

Re: Differences between the word2vec paper and its implementation

#71

Post by b_tterc_p » Tue, Jun 04, 2019, 2:42 AM UTC

gojomo wrote: post #55

b_tterc_p wrote: post #19

On a similar note, a long time ago I read the Doc2Vec paper, then looked at popular Doc2Vec implementations. They didn’t seem to do the same thing. The paper said you basically make vectors for words, then append on an additional space that represents the additional information of documents as opposed to single words. All popular implementations I found seemed to put the document vectors into the same space as the wo…

The only code released by the 'Paragraph Vector' paper authors was a small patch, from Mikolov, that added paragraph-vectors to the original `word2vec.c` implementation in a very simple way: treating the 1st token of each line as a special paragraph-vector, still string-named (and allocated in the same lookup dictionary). Only by convention (a special prefix on those paragraph-vector tokens) could collisions with sim…

Insightful. Thanks

Re: Differences between the word2vec paper and its implementation

#72

Post by throwawayjava » Tue, Jun 04, 2019, 2:46 AM UTC

jfoutz wrote: post #69

throwaway287391 wrote: post #63

Earlier quoted context omitted.

I don't have much to say to most of this, but I'll point out that I didn't and couldn't have downvoted you as none of my throwaways have >=500 karma or whatever the threshold is.

> I'll point out that I didn't and couldn't have downvoted you as none of my throwaways have >=500 karma I'll take you at your word. You want to just drop in and drop bombs and pretend that the information you are presenting is in some sense useful. Other folks pointed out that you are wrong. But you also want the reputation. You want the respect for revising your opinion. You're you from your perspective, but from e…

Man, I hate when I get dragged into these sorts of intense emotional downs by silly internet conversations with strangers.

FWIW I undowned your posts; internet karma is about communication/moderation, and sometimes has the opposite of the intended effect. I've been there.

Anyways, find some time to pet a dog/cat or fly a kite today.

Re: Differences between the word2vec paper and its implementation

#73

Post by throwawayjava » Tue, Jun 04, 2019, 2:50 AM UTC

p1esk wrote: post #58

liquidify wrote: post #38

Earlier quoted context omitted.

If there are bugs in your code that would have prevented you from being able to do what you claim to do, then to the readers of your paper, it seems more likely that you might not have performed the experimentation that you claim to.

Typically when I post code for my papers, I try to clean it up, because it's an ugly mess that is hard to understand (even myself sometimes). This clean up process might introduce bugs. I'm not going to thoroughly test everything, I have next paper deadline coming up and my adviser could not care less about my code. But, if you're interested in my research and having trouble reproducing my results, email me and I'll…

...and university/company sometimes makes you wipe out the git commit log, so you can't always just use "HEAD at time of submission".

Pro tip: at paper submission time, md5sum your code and also git tag it in your private repo. When you release the code, if you have to / want to release with a clean history, make submission-time your initial commit and then make the current state of the repo your second commit. I've never encountered an institution that won't allow that level of history in the code release, even places that are pretty hard core about wiping pre-release history.

Re: Differences between the word2vec paper and its implementation

#74

Post by Volt » Tue, Jun 04, 2019, 2:54 AM UTC

randcraw wrote: post #45

liquidify wrote: post #38

Earlier quoted context omitted.

If there are bugs in your code that would have prevented you from being able to do what you claim to do, then to the readers of your paper, it seems more likely that you might not have performed the experimentation that you claim to.

Agreed. If your results are not reproducible by others, the publication should issue a retraction and withdraw your work. Code that works is essential.

I hope you're a scientist.

Re: Differences between the word2vec paper and its implementation

#75

Post by jonas21 » Tue, Jun 04, 2019, 3:00 AM UTC

jfoutz wrote: post #53

throwaway287391 wrote: post #49

Earlier quoted context omitted.

You're right, I misread that part.

throwaway287391 it's hilarious how much you value your own opinion. You feel it's important your interlocutor knows you know you were corrected, while remaining anonymous. This is like, the funniest thing i've seen all week. You should run for office. _edit_ Mmmm delicious downvotes. How many alts do you have throwaway287391? i don't think dang would or should expose you. But it is even more funny to me that you woul…

FWIW, I downvoted because because you somehow managed to break about half of the HN comment guidelines [1] in a single comment, in particular:

* Be kind. Don't be snarky. Comments should get more thoughtful and substantive, not less, as a topic gets more divisive.

* When disagreeing, please reply to the argument instead of calling names. "That is idiotic; 1 + 1 is 2, not 3" can be shortened to "1 + 1 is 2, not 3."

* Please respond to the strongest plausible interpretation of what someone says, not a weaker one that's easier to criticize. Assume good faith.

* Please don't make insinuations about astroturfing. It degrades discussion and is usually mistaken. If you're worried, email us and we'll look at the data.

* Please don't comment about the voting on comments. It never does any good, and it makes boring reading.

[1] https://news.ycombinator.com/newsguidelines.html

Re: Differences between the word2vec paper and its implementation

#76

Post by caenorst » Tue, Jun 04, 2019, 3:01 AM UTC

p1esk wrote: post #58

liquidify wrote: post #38

Earlier quoted context omitted.

If there are bugs in your code that would have prevented you from being able to do what you claim to do, then to the readers of your paper, it seems more likely that you might not have performed the experimentation that you claim to.

Typically when I post code for my papers, I try to clean it up, because it's an ugly mess that is hard to understand (even myself sometimes). This clean up process might introduce bugs. I'm not going to thoroughly test everything, I have next paper deadline coming up and my adviser could not care less about my code. But, if you're interested in my research and having trouble reproducing my results, email me and I'll…

And don't you think it would have been more efficient to write it down clean at first with good comments and code practice ? Those good practices are here for a reason and not just for annoying you for the sack of "beauty". You would save more time for yourself when you wanna expand the paper further or just reuse some part of it, have better reputation for your work, and gain some citation because people will have the tendency to use your code as a starting point.

Re: Differences between the word2vec paper and its implementation

#77

Post by utopcell » Tue, Jun 04, 2019, 3:10 AM UTC

bollu wrote: post #29

utopcell wrote: post #27

I think it very unfair to the original set of word2vec papers to be talking about 'academic dishonesty'. This is a case of a user that has little to no experience with neural networks. There are a ton of articles describing the need for random initialization [1][2]. In fact, if one spends a few seconds thinking about it, the need is evident. Without it, the NN cannot perform symmetry breaking: If inputs are set to ze…

That's hardly the point of the article --- the actual paper does not describe the use of two separate vectors for each word. The initialization was an interesting tidbit.

Of course it does. For word w they are v_w and v'_w on Eq.4 for the case of SGNS [1].

[1] https://papers.nips.cc/paper/5021-distributed-representation...

Re: Differences between the word2vec paper and its implementation

#78

Post by utopcell » Tue, Jun 04, 2019, 3:16 AM UTC

eggie5 wrote: post #37

instead of thinking about what it is in practice: skip-gram negative sampling, I think it's much more intuitive to think about what it is in theory: extreme multi-class classification. word2vec is a multi-class classification problem with a softmax output layer and cross-entropy loss. The novel part of word2vec, in my opinion, is two: 1. dataset (proximal input word & output word) generation from documents eg: skiagr…

Can't really call it a speedup trick, since it actually improves the performance of the embeddings but in terms of qualitative understanding, I see where you're coming from.

Re: Differences between the word2vec paper and its implementation

#79

Post by inimino » Tue, Jun 04, 2019, 3:21 AM UTC

caenorst wrote: post #76

p1esk wrote: post #58

Earlier quoted context omitted.

Typically when I post code for my papers, I try to clean it up, because it's an ugly mess that is hard to understand (even myself sometimes). This clean up process might introduce bugs. I'm not going to thoroughly test everything, I have next paper deadline coming up and my adviser could not care less about my code. But, if you're interested in my research and having trouble reproducing my results, email me and I'll…

And don't you think it would have been more efficient to write it down clean at first with good comments and code practice ? Those good practices are here for a reason and not just for annoying you for the sack of "beauty". You would save more time for yourself when you wanna expand the paper further or just reuse some part of it, have better reputation for your work, and gain some citation because people will have t…

"Writing it down clean at first" implies that you know what is going to work when you start. If that's the case, it's not research.

Re: Differences between the word2vec paper and its implementation

#80

Post by DannyBee » Tue, Jun 04, 2019, 3:24 AM UTC

bollu wrote: post #25

DannyBee wrote: post #17

Speaking as someone who has read about 40 years of papers in compiler optimization, it's very interesting. In the early days, there were fairly exact algorithms that worked as described, and were implemented as described, but were pseudocoded in papers. Where the pseudocode differed from implementation, differences were described in great detail (IE they may say an array can be shared but isn't to make the pseudocode…

As someone who works on compiler optimisations, I'd love to get a link the Rice compiler's source code --- do you happen to have a link?

Not sure i still have it around ping me at the email in my profile and i'll see what i can do. It used to be on their ftp server, which they took down.

I expect if you email ken kennedy over at rice, he may be able to find it as well.