Differences between the word2vec paper and its implementation • Hacker News Board

Re: Differences between the word2vec paper and its implementation

#121

Post by Dylan16807 » Tue, Jun 04, 2019, 11:30 AM UTC

visarga wrote: post #95

KirinDave wrote: post #51

Earlier quoted context omitted.

Pardon me, maybe I just misunderstood the whole idea of research but what good is it if it's not reproducible? I can understand it may be part of a meaningful personal journey for you, and I appreciate that. But if no one else can validate your research they're correct to discredit it and you. So what is the optimal outcome here? Should we hold you to a standard of reproducibility even if it is as minimal as, "actual…

Related to w2v, the proof is in the vectors. They work. And, by the way, w2v has had a whole family of variants which were built on top of the original implementation (such as Doc2Vec, Doc2VecC and others). Related to most ML code, it's not written in C but in Python frameworks that implement whole layers as building blocks. So it's not as hard to read/change. You can take something complex like the transformer and u…

For almost all ML training, it's straightforward to break it into a fixed number of chunks without race conditions.

That's if float rounding issues are even a big problem in the first place. If your results are within .1% over a few runs it's reproducible enough for most purposes.

Re: Differences between the word2vec paper and its implementation

#122

Post by izacus » Tue, Jun 04, 2019, 11:43 AM UTC

nullc wrote: post #82

billconan wrote: post #6

For the past week I have been frustrated by an opensource code of a deep learning paper. This type of things are so common in academia. The particular code I looked at has missing documentation, hardcoded local paths, broken dataset download links and broken pretrained model download links. I have to fix bugs before the code can run. I'm very curious how did the author run that code with the bugs. I call them insince…

There appears to be something of a software engineering crisis in academia. Imagine that you are a graduate student or a newly minted PHD who happens to be pretty good at practical software engineering: You can leave academia for FAANG and a solid six figure income, or you can stay struggling, poorly paid, unable to get tenure in academia. In some groups software engineers are treated as glorified typists, too valuab…

> Another kind of fake result I've encountered which is even less directly a result of the software engineering shortage is in signal processing literature. While working on audio/video compression I found it common for algorithms to be presented without various constants and after reimplementing and asking the authors for their constants I found that they'd been cherrypicked for the ten images used in the paper, and that the whole approach doesn't actually work. This is a kind of ineptitude (or outright dishonesty) that would be much less common in a world where reviewers received a working and usable implementation in source form-- but that can't be expected in a world where qualified software engineering is not readily available to researchers.

This happened so many times when I was working on NLP/CV algorithms - I've read and implemented many algorithms from papers just to find out that they only produce the amazing improvements on cherrypicked dataset. On pretty much all other practical data the algorithms performed worse and in many cases even crashed!

Re: Differences between the word2vec paper and its implementation

#123

Post by izacus » Tue, Jun 04, 2019, 11:46 AM UTC

newen wrote: post #2

This kind of things happens all the time in academia. The authors are either constrained by space due to paper limitations or they are too lazy to explain all the little details that go into the algorithm. I used to do research in computer vision a few years ago and it used to be that people won't publish their code and they purposely won't put in all of the details of the algorithm in the paper. Many of those algori…

Even worse, when actually researching CV, it seems that leaving out the details can even be deliberate in papers - to hide the fact that the authors tuned and cherry picked input datasets so they show the improvements they claim. Actually implementing the code and running on more standard datasets tends to quickly put results of many papers into question.

Re: Differences between the word2vec paper and its implementation

#124

Post by slx26 » Tue, Jun 04, 2019, 11:50 AM UTC

Just as a curiosity, complementing what others have already written... I read part of Mikolov's thesis and code in the past (when I was still studying at the university, so I might have got everything wrong (I still don't get half of it :D)). First I found it quite shocking that the code was so bad. The training code was pretty confusing to me, and I found the lack of useful comments discouraging. The test code (which loaded stored embeddings from a file and allowed some basic operations) was even much, much worse. Like, declaring three variables (a, b, c) and reusing them for different things in the main functions without explaining anything, and doing linear searches through the whole embeddings to find a word vector... very ugly and scary things.

So, I had a very bad impression of the code. But then, I checked the thesis, and I found it awesome. The amount of tests and implementations the guy made, and how he showed in practice how better results could be achieved in a good number of different setups... I found it really impressive. But such great work paired with such bad code! I was just a CS student, so I found it shocking. Nowadays I realize he was simply focused on a different thing, and the results he obtained were indeed outstanding and talk for themselves.

It's easy to look back and criticise the code, but when you look at the work he did in perspective... it's completely unfair to ask more from him (admittedly, they had time to address some of the issues later, but they probably had better things to do too).

Re: Differences between the word2vec paper and its implementation

#125

Post by seanmcdirmid » Tue, Jun 04, 2019, 12:49 PM UTC

opportune wrote: post #89

seanmcdirmid wrote: post #56

Earlier quoted context omitted.

A lot of CS papers aren’t testing algorithms at all.

It is very common in the last few pages of a CS paper to benchmark your results against other algorithms or models

And it’s also very common not to. CS papers are pretty diverse in what they are talking about, many aren’t algorithm papers.

Re: Differences between the word2vec paper and its implementation

#126

Post by duckmysick » Tue, Jun 04, 2019, 1:25 PM UTC

opportune wrote: post #87

bayareanative wrote: post #84

Earlier quoted context omitted.

Editors gotta be more rigorous and only accept papers with completely reproducible portable examples, i.e., docker images, literate code and source code repos. Pseudocode is helpful to be platform neutral, but if it's not precise enough to implemented as code, then it's still a proprietary figment of someone else's imagination akin to the squishy social sciences where almost anything goes, not rigorously reproducible…

In some fields, like in ML/AI or in other data-sciencey fields, keeping your code / training data closed prevents other researchers from building or improving on your work. It's more than just monetization, in that case it's just tragedy-of-the-commons career growth

> keeping your code / training data closed prevents other researchers from building or improving on your work.

I might be naive in this regard, but isn't that the main point of doing research?

Re: Differences between the word2vec paper and its implementation

#127

Post by IfOnlyYouKnew » Tue, Jun 04, 2019, 1:47 PM UTC

billconan wrote: post #6

For the past week I have been frustrated by an opensource code of a deep learning paper. This type of things are so common in academia. The particular code I looked at has missing documentation, hardcoded local paths, broken dataset download links and broken pretrained model download links. I have to fix bugs before the code can run. I'm very curious how did the author run that code with the bugs. I call them insince…

https://github.com/jonadsimon/entendrepreneur-web?

Re: Differences between the word2vec paper and its implementation

#128

Post by dekhn » Tue, Jun 04, 2019, 1:49 PM UTC

duckmysick wrote: post #126

opportune wrote: post #87

Earlier quoted context omitted.

In some fields, like in ML/AI or in other data-sciencey fields, keeping your code / training data closed prevents other researchers from building or improving on your work. It's more than just monetization, in that case it's just tragedy-of-the-commons career growth

> keeping your code / training data closed prevents other researchers from building or improving on your work. I might be naive in this regard, but isn't that the main point of doing research?

Sure, but the challenge is, if you're in a competitive field and you make it easy for your competitors, you're going to have a harder time establishing job security. Currently many fields like biology, CS, ML, physics have fewer well-paying job positions than contenders, so every competitive advantage makes a difference. I know plenty of professors who got there by carefully writing papers that lacked specific details on how to reproduce a cutting-edge, "hot" paper.

Re: Differences between the word2vec paper and its implementation

#129

Post by taeric » Tue, Jun 04, 2019, 1:50 PM UTC

billconan wrote: post #22

liuliu wrote: post #13

Earlier quoted context omitted.

Maintaining a pipeline that works takes effort that can be spent somewhere else. As long as it can be reproduced with a little effort from me, I really don't mind. I would rather they spend time to try a lot of different things, build intuitions, and search better backbone structures than maintaining a CI system so that every time their code updates can train imageNet to That should be left to TensorFlow model zoo or…

I'm not asking for much, but the code should be runnable at least. This is why I think jupyter notebook and things alike are so important as the future publishing media. Reproducibility is very important.

Jupyter notebooks are laughably not reproducible right now. They may get there, but they encourage bad development habits. Actively.

Re: Differences between the word2vec paper and its implementation

#130

Post by DannyBee » Tue, Jun 04, 2019, 2:42 PM UTC

ms013 wrote: post #96

DannyBee wrote: post #80

Earlier quoted context omitted.

Not sure i still have it around ping me at the email in my profile and i'll see what i can do. It used to be on their ftp server, which they took down. I expect if you email ken kennedy over at rice, he may be able to find it as well.

Ken died like 15+ years ago. Contact John Mellor-Crummey instead: he still does compiler/language work there AFAIK.

Sorry, yes, Ken passed away, i meant keith cooper, i was writing quickly.