Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Differences between the word2vec paper and its implementation

github.com

Post reply on HN
Print view View original

111–120 of 156 posts

Re: Differences between the word2vec paper and its implementation

#111

Post by elcomet »

sampo wrote: post #107
elcomet wrote: post #102

Earlier quoted context omitted.

It's not the editor's job to replicate the results. Imagine the same requirements in medical research, this would be crazy. Editors accept a paper based on the quality of the paper and the claim. Then the paper needs to be replicated a few times by independent research teams to be considered valid. This is the step we are probably missing in AI. Any published paper (and even unpublished, on arxiv) are considered vali…

> It's not the editor's job to replicate the results. They obviously meant the reviewers. Not everyone is intimately familiar with academic terminology.

The reviewers are commissioned by the editor, so I would say it's the same thing. Unless you meant the reviewers should replicate the papers with their own ressources, but I think it's not what we want. Reviewers are benevolent, and they don't have time nor resources to replicate results.

Maybe a solution would be a platform, like a CI for machine learning where authors would send their codes, and the CI would run it for them, and make the results public. Then everyone could check the code and results, reviewers included.

Re: Differences between the word2vec paper and its implementation

#112

Post by dTal »

colechristensen wrote: post #12
billconan wrote: post #6

For the past week I have been frustrated by an opensource code of a deep learning paper. This type of things are so common in academia. The particular code I looked at has missing documentation, hardcoded local paths, broken dataset download links and broken pretrained model download links. I have to fix bugs before the code can run. I'm very curious how did the author run that code with the bugs. I call them insince…

It is very common. Scientist code isn't software engineering code (and software engineering code is very often worthy of criticism) It's just good news when you can find source code at all instead of just being told vague things about something the author did.

>It's just good news when you can find source code at all instead of just being told vague things about something the author did.

This is extremely common, and I don't see why it should be allowed. I could easily write a paper claiming some kind of interesting result, fake a few "ground truth vs my algorithm" pairs, and talk a bit of mathy-sounding piffle about what it does - and it would be indistinguishable from many papers I have read.

In my opinion - provide runnable source, or what you have isn't a paper, it's a boast.

Re: Differences between the word2vec paper and its implementation

#113

Post by bollu »

DannyBee wrote: post #80
bollu wrote: post #25

Earlier quoted context omitted.

As someone who works on compiler optimisations, I'd love to get a link the Rice compiler's source code --- do you happen to have a link?

Not sure i still have it around ping me at the email in my profile and i'll see what i can do. It used to be on their ftp server, which they took down. I expect if you email ken kennedy over at rice, he may be able to find it as well.

You seem to have taken down your e-mail information on your profile :)

Re: Differences between the word2vec paper and its implementation

#114

Post by amelius »

nullc wrote: post #82
billconan wrote: post #6

For the past week I have been frustrated by an opensource code of a deep learning paper. This type of things are so common in academia. The particular code I looked at has missing documentation, hardcoded local paths, broken dataset download links and broken pretrained model download links. I have to fix bugs before the code can run. I'm very curious how did the author run that code with the bugs. I call them insince…

There appears to be something of a software engineering crisis in academia. Imagine that you are a graduate student or a newly minted PHD who happens to be pretty good at practical software engineering: You can leave academia for FAANG and a solid six figure income, or you can stay struggling, poorly paid, unable to get tenure in academia. In some groups software engineers are treated as glorified typists, too valuab…

I think there should be more software engineering in academia for different reasons too.

In the past academia and government agencies developed the internet and its communication protocols; this was (imho) a much better internet than the corporation-dominated internet we have today.

I'd like to see a resurge of this type of internet. Let us use well-researched open protocols again; and let us use applications which are really built for people, not advertisers. Let corporations build the hardware, but let us keep our data far away from them.

And besides the computer science / software engineering branches, I'd like to see other fields (like industrial design, and for instance even sociology) to join the development of a better digital future. The kind of future where everybody profits, not just shareholders of big companies.

Re: Differences between the word2vec paper and its implementation

#115

Post by krageon »

billconan wrote: post #6

For the past week I have been frustrated by an opensource code of a deep learning paper. This type of things are so common in academia. The particular code I looked at has missing documentation, hardcoded local paths, broken dataset download links and broken pretrained model download links. I have to fix bugs before the code can run. I'm very curious how did the author run that code with the bugs. I call them insince…

I had the great displeasure to work on multiple state-of-the-art algorithms related to sticking things in buckets with varying degrees of optimisation and/or supervision. There are a ton of reasonably recent papers for that, some of them have source code (most of them have nigh-impenetrable theory on why their algorithm is the best) and every case I tested was emphatically much worse than the technologies we've been using for 10+ years in any practical sense.

That problem is compounded with the issues you describe, which I'd categorise as probably the best case. I did not find even a single case of a properly opensourced repository that just compiled somewhere that wasn't the researcher's laptop.

Re: Differences between the word2vec paper and its implementation

#116

Post by anewhnaccount2 »

p1esk wrote: post #93
Qworg wrote: post #92

Earlier quoted context omitted.

The editors, previous to accepting. If it doesn't run out of the box and reproduce the result, it doesn't get published.

Some code might require 8 GPUs and a week to verify a single result. There could be a dozen results to be verified in a paper. Are you sure you have thought this through?

The experiments themselves should be conducted on fully logged public infrastructure so that it impossible to miss anything out. This final log would include the inputs: shell commands/git repositories/Dockerfiles, training and testing data; important intermediate data: Docker images and trained models; and outputs: tables of numbers or what have you. This way, the whole thing is vouched for automatically, and the editors don't need to do anything.

It would be a big change, but this is actually possible, and it would make things easier for everyone down the line.

Re: Differences between the word2vec paper and its implementation

#117

Post by gdy »

bayareanative wrote: post #84
DannyBee wrote: post #17

Speaking as someone who has read about 40 years of papers in compiler optimization, it's very interesting. In the early days, there were fairly exact algorithms that worked as described, and were implemented as described, but were pseudocoded in papers. Where the pseudocode differed from implementation, differences were described in great detail (IE they may say an array can be shared but isn't to make the pseudocode…

Editors gotta be more rigorous and only accept papers with completely reproducible portable examples, i.e., docker images, literate code and source code repos. Pseudocode is helpful to be platform neutral, but if it's not precise enough to implemented as code, then it's still a proprietary figment of someone else's imagination akin to the squishy social sciences where almost anything goes, not rigorously reproducible…

"Sometimes I think some researchers think they're helping themselves keep their research proprietary so they will able to monetize their special knowledge or implementation, especially if no one else can make it work ("knowledge" (job) security/silo)."

My very limited experience says it is true. While our team worked for a commercial company in a certain highly competetive industry, I've both read many papers describing algos that on a closer look couldn't work as described and myself was forbidden to publish anything allowing our competitors or potential buyers to reproduce our algorithms. Our team ended up submitting a paper that was turned down by a reviewer and called an advertisement for the lack of details. None of us wanted to 'fix' that by adding fake details as in the papers we had read, so we ended up not publishing anything while working on the projects in that industry.

Re: Differences between the word2vec paper and its implementation

#118

Post by cfcf14 »

patrick5415 wrote: post #103
KirinDave wrote: post #51

Earlier quoted context omitted.

Pardon me, maybe I just misunderstood the whole idea of research but what good is it if it's not reproducible? I can understand it may be part of a meaningful personal journey for you, and I appreciate that. But if no one else can validate your research they're correct to discredit it and you. So what is the optimal outcome here? Should we hold you to a standard of reproducibility even if it is as minimal as, "actual…

Reproducible research does not mean you don’t have to do any work. The original author isn’t going to come to your lab and clean your pipettes nor should they. Grow up.

The research published in this case (the original word2vec paper) is quite literally not what it claims to be. It is factually incorrect. Is that really the responsibility of the individual attempting to reproduce the work to identify and correct?

How wrong should the original work be before it ceases to be a case of 'clean your own pipettes'?

Re: Differences between the word2vec paper and its implementation

#119

Post by itronitron »

colechristensen wrote: post #12
billconan wrote: post #6

For the past week I have been frustrated by an opensource code of a deep learning paper. This type of things are so common in academia. The particular code I looked at has missing documentation, hardcoded local paths, broken dataset download links and broken pretrained model download links. I have to fix bugs before the code can run. I'm very curious how did the author run that code with the bugs. I call them insince…

It is very common. Scientist code isn't software engineering code (and software engineering code is very often worthy of criticism) It's just good news when you can find source code at all instead of just being told vague things about something the author did.

In my opinion, a paper should describe the contribution in sufficient detail that it can be implemented by someone with similar expertise. I believe that would foster greater innovation as people would necessarily have greater familiarity with all of the details, and they would better see why the thing works or doesn't.

Re: Differences between the word2vec paper and its implementation

#120

Post by bjourne »

nullwasamistake wrote: post #39
DannyBee wrote: post #17

Speaking as someone who has read about 40 years of papers in compiler optimization, it's very interesting. In the early days, there were fairly exact algorithms that worked as described, and were implemented as described, but were pseudocoded in papers. Where the pseudocode differed from implementation, differences were described in great detail (IE they may say an array can be shared but isn't to make the pseudocode…

I ran into the same thing when implementing a hash table sorting algorithm from a recent paper. Nowhere in the paper does it mention that table size MUST be a power of two. The new table structure is supposedly more memory efficient than anything else out there. That's the whole point of it. So it's nice that they leave out "btw only works with size power of two". I got 2/3 of the way through an optimized implementat…

Backing arrays for fast hash tables are always powers of 2. That's how they are implemented in all mainstream languages. Details are often omitted in academia because of publishing constraints or because the authors thought they were obvious.
Post reply on HN