Differences between the word2vec paper and its implementation • Hacker News Board

Re: Differences between the word2vec paper and its implementation

#91

Post by inimino » Tue, Jun 04, 2019, 5:18 AM UTC

ur-whale wrote: post #66

YeGoblynQueenne wrote: post #35

Earlier quoted context omitted.

In machine learning, you don't get credit for publishing rigorous papers. You get credits for publishing papers that show improved performance: One big challenge the community faces is that if you want to get a paper published in machine learning now it's got to have a table in it, with all these different data sets across the top, and all these different methods along the side, and your method has to look like the b…

>It's an extremely, almost exclusively, empirical field. Fully agree, and it's a necessary disease in young fields like ML (akin to grid search in fact). But at some point, there will need to be some sort of theoretical foundation brought to bear or advancement will grind down to a halt And the academic reward mechanism needs to start reflecting that fact.

I've been quite disappointed to see how many papers from "cutting edge" research groups are chasing small improvements in well-known benchmarks by finding new techniques that happen to work, while there is a lot less effort put into finding out why. I guess Geoff Hinton's explanation about what gets published today explains it.

Re: Differences between the word2vec paper and its implementation

#92

Post by Qworg » Tue, Jun 04, 2019, 5:33 AM UTC

p1esk wrote: post #90

bayareanative wrote: post #84

Earlier quoted context omitted.

Editors gotta be more rigorous and only accept papers with completely reproducible portable examples, i.e., docker images, literate code and source code repos. Pseudocode is helpful to be platform neutral, but if it's not precise enough to implemented as code, then it's still a proprietary figment of someone else's imagination akin to the squishy social sciences where almost anything goes, not rigorously reproducible…

And who exactly is supposed to verify that code is reproducible for all the papers that get accepted?

The editors, previous to accepting. If it doesn't run out of the box and reproduce the result, it doesn't get published.

Re: Differences between the word2vec paper and its implementation

#93

Post by p1esk » Tue, Jun 04, 2019, 5:49 AM UTC

Qworg wrote: post #92

p1esk wrote: post #90

Earlier quoted context omitted.

And who exactly is supposed to verify that code is reproducible for all the papers that get accepted?

The editors, previous to accepting. If it doesn't run out of the box and reproduce the result, it doesn't get published.

Some code might require 8 GPUs and a week to verify a single result. There could be a dozen results to be verified in a paper.

Are you sure you have thought this through?

Re: Differences between the word2vec paper and its implementation

#94

Post by tsbinz » Tue, Jun 04, 2019, 5:56 AM UTC

p1esk wrote: post #93

Qworg wrote: post #92

Earlier quoted context omitted.

The editors, previous to accepting. If it doesn't run out of the box and reproduce the result, it doesn't get published.

Some code might require 8 GPUs and a week to verify a single result. There could be a dozen results to be verified in a paper. Are you sure you have thought this through?

8 gpus and a week is cheap even, there are popular papers that need upwards of 10k GPU hours.

Re: Differences between the word2vec paper and its implementation

#95

Post by visarga » Tue, Jun 04, 2019, 5:59 AM UTC

KirinDave wrote: post #51

throwaway287391 wrote: post #30

Earlier quoted context omitted.

This sort of thing is aggravating to read. Frankly it comes off as really entitled. As researchers, the expectation is now that we not only have to do the research and write a paper like the good old days, but we have to release the code too. Okay, fine. But now that's not enough either -- the code has to be well-documented and clean. Ugh, alright, fine -- it's going to take me a few extra weeks of not doing research…

Pardon me, maybe I just misunderstood the whole idea of research but what good is it if it's not reproducible? I can understand it may be part of a meaningful personal journey for you, and I appreciate that. But if no one else can validate your research they're correct to discredit it and you. So what is the optimal outcome here? Should we hold you to a standard of reproducibility even if it is as minimal as, "actual…

Related to w2v, the proof is in the vectors. They work. And, by the way, w2v has had a whole family of variants which were built on top of the original implementation (such as Doc2Vec, Doc2VecC and others).

Related to most ML code, it's not written in C but in Python frameworks that implement whole layers as building blocks. So it's not as hard to read/change. You can take something complex like the transformer and understand it in a few minutes. The most puzzling part with framework code is figuring out the shape of tensors and, sometimes, what each dimension means.

Regarding reproducibility - it's hard to achieve on account of parallelisation. You see, neural nets use floats, and floats are not real numbers. Half the float numbers lie between [-1, 1] and the rest outside. If you do 1e-10+1e10 you get 1e10. So if you add 1000 times 1e-10 to 1e+10 you still get 1e+10, but if you add them up together first, the result is different. So float summation is not commutative. Depending on race conditions the order in which summation occurs could change and the result change as well, even if you use the same random seeds. And if we give up parallelism then we can't run the experiments any more.

Re: Differences between the word2vec paper and its implementation

#96

Post by ms013 » Tue, Jun 04, 2019, 6:04 AM UTC

DannyBee wrote: post #80

bollu wrote: post #25

Earlier quoted context omitted.

As someone who works on compiler optimisations, I'd love to get a link the Rice compiler's source code --- do you happen to have a link?

Not sure i still have it around ping me at the email in my profile and i'll see what i can do. It used to be on their ftp server, which they took down. I expect if you email ken kennedy over at rice, he may be able to find it as well.

Ken died like 15+ years ago. Contact John Mellor-Crummey instead: he still does compiler/language work there AFAIK.

Re: Differences between the word2vec paper and its implementation

#97

Post by karlding » Tue, Jun 04, 2019, 6:10 AM UTC

DannyBee wrote: post #17

Speaking as someone who has read about 40 years of papers in compiler optimization, it's very interesting. In the early days, there were fairly exact algorithms that worked as described, and were implemented as described, but were pseudocoded in papers. Where the pseudocode differed from implementation, differences were described in great detail (IE they may say an array can be shared but isn't to make the pseudocode…

What if you simply worked on the AST/token level and converted that into pseudocode, assuming you either annotated your functions with a description or named them following some sort of convention? Then you could write your code and then "compile" your source files to pseudocode.

Seems like this could be useful for writing specification documents or anything safety critical..

Re: Differences between the word2vec paper and its implementation

#98

Post by raverbashing » Tue, Jun 04, 2019, 6:31 AM UTC

colechristensen wrote: post #12

billconan wrote: post #6

For the past week I have been frustrated by an opensource code of a deep learning paper. This type of things are so common in academia. The particular code I looked at has missing documentation, hardcoded local paths, broken dataset download links and broken pretrained model download links. I have to fix bugs before the code can run. I'm very curious how did the author run that code with the bugs. I call them insince…

It is very common. Scientist code isn't software engineering code (and software engineering code is very often worthy of criticism) It's just good news when you can find source code at all instead of just being told vague things about something the author did.

> Scientist code isn't software engineering code (and software engineering code is very often worthy of criticism)

Absolutely correct, and not in a good way

Though to be fair, people doing research have other interests besides maintainable code (though half or more of the annoyances will probably come and bite them later)

Re: Differences between the word2vec paper and its implementation

#99

Post by hoseja » Tue, Jun 04, 2019, 6:37 AM UTC

throwaway287391 wrote: post #30

billconan wrote: post #6

For the past week I have been frustrated by an opensource code of a deep learning paper. This type of things are so common in academia. The particular code I looked at has missing documentation, hardcoded local paths, broken dataset download links and broken pretrained model download links. I have to fix bugs before the code can run. I'm very curious how did the author run that code with the bugs. I call them insince…

This sort of thing is aggravating to read. Frankly it comes off as really entitled. As researchers, the expectation is now that we not only have to do the research and write a paper like the good old days, but we have to release the code too. Okay, fine. But now that's not enough either -- the code has to be well-documented and clean. Ugh, alright, fine -- it's going to take me a few extra weeks of not doing research…

Your paper has to be well documented and clean. The code is often more important than your paper.

Re: Differences between the word2vec paper and its implementation

#100

Post by visarga » Tue, Jun 04, 2019, 6:44 AM UTC

woliveirajr wrote: post #81

b_tterc_p wrote: post #19

On a similar note, a long time ago I read the Doc2Vec paper, then looked at popular Doc2Vec implementations. They didn’t seem to do the same thing. The paper said you basically make vectors for words, then append on an additional space that represents the additional information of documents as opposed to single words. All popular implementations I found seemed to put the document vectors into the same space as the wo…

This. And my thesis was about how tf-idf and embeddings with deep (deeeeeep) neural networks could be give better results in authorship attribution.

> embeddings with deep (deeeeeep) neural networks

Embeddings are shallow, and what comes after them is usually less than 5 layers of LSTM, not a deeeeeep neural net (maybe, deep only horizontally, on the axis of words).