It’s patented BTW — https://patents.google.com/patent/US9037464B1/en
Turns the patent only describes the paper, but not the implementation. Great, and somewhat ironic
Differences between the word2vec paper and its implementation
141–150 of 156 posts
Re: Differences between the word2vec paper and its implementation
#142Earlier quoted context omitted.
In some fields, like in ML/AI or in other data-sciencey fields, keeping your code / training data closed prevents other researchers from building or improving on your work. It's more than just monetization, in that case it's just tragedy-of-the-commons career growth
> keeping your code / training data closed prevents other researchers from building or improving on your work. I might be naive in this regard, but isn't that the main point of doing research?
In theory, yes. The actual main point of doing research, from the researcher's point of view, is to advance their career. Typically this means posting cutting-edge results in high impact journals using novel methods. Relinquishing control over crucial details allowing a researcher to continue publishing high-impact papers would increase the competition for the researcher in that academic space, making it harder for that individual to advance their career.
Re: Differences between the word2vec paper and its implementation
#143Earlier quoted context omitted.
> keeping your code / training data closed prevents other researchers from building or improving on your work. I might be naive in this regard, but isn't that the main point of doing research?
In theory, yes. The actual main point of doing research, from the researcher's point of view, is to advance their career. Typically this means posting cutting-edge results in high impact journals using novel methods. Relinquishing control over crucial details allowing a researcher to continue publishing high-impact papers would increase the competition for the researcher in that academic space, making it harder for t…
How do the sources of funding (R&D divisions, universities, governments) fit in this picture? Do they have access to the "secret sauce" or do they have to pay extra consultation fees on top of research grants.
Re: Differences between the word2vec paper and its implementation
#144Earlier quoted context omitted.
Please keep in mind that currently there are a lot of unqualified people reviewing papers, even at top conferences, as evidenced by OpenReview.net. There was an experiment done at NIPS a few years ago where something like half of the rejected papers got accepted after having been reviewed independently by different reviewers, and/or the other way around. If people can't even agree if paper is good or not, while assum…
There is a difference between unqualified people, and qualified but busy people. While there may be unqualified people reviewing papers, even qualified people who don't put in enough time and effort to review a paper comes to the same result. But the solution for that problem will look different. There's also a big difference between accepting a paper and judging if it is good . At a top conference like NIPS, lots of…
Submitting a paper to a conference for it to be published 11 months later is not going to work - the paper will be hopelessly obsolete by then.
A better solution imo is to improve openreview.net process somehow, so that I'm motivated to go there, find papers relevant to my research (ideally get notified when such papers are posted for review), and leave a review (and perhaps vote on other reviews just like we do here on HN, or do something else to influence the acceptance decision). Obviously there should be methods to prevent abuse: moderators, reputations of reviewers, restricting reviewers to those with specific publications (e.g. based on keywords), etc. Btw, code reproducibility could be used to give some weight in such public review process.
Re: Differences between the word2vec paper and its implementation
#145Earlier quoted context omitted.
Some code might require 8 GPUs and a week to verify a single result. There could be a dozen results to be verified in a paper. Are you sure you have thought this through?
It is not like, ehem, Elsevier does not have the money to pay for this...
AI papers do not get published by Elsevier
Re: Differences between the word2vec paper and its implementation
#146Earlier quoted context omitted.
Some code might require 8 GPUs and a week to verify a single result. There could be a dozen results to be verified in a paper. Are you sure you have thought this through?
The experiments themselves should be conducted on fully logged public infrastructure so that it impossible to miss anything out. This final log would include the inputs: shell commands/git repositories/Dockerfiles, training and testing data; important intermediate data: Docker images and trained models; and outputs: tables of numbers or what have you. This way, the whole thing is vouched for automatically, and the ed…
Who is going to pay for that?
Re: Differences between the word2vec paper and its implementation
#147Earlier quoted context omitted.
Pardon me, maybe I just misunderstood the whole idea of "reproducible" but it is only not reproducible if you tried to implement the ideas described in the paper yourself, and I mean really tried, and contacted the authors for help, and still failed to achieve the claimed results.
> if you tried to implement the ideas described in the paper yourself, and I mean really tried, and contacted the authors for help, and still failed to achieve the claimed results. This is the actually important definition of "reproducible". "has an install.sh" is really nice, but it's far more important that an informed reader can recreate the artifact for themselves from the written description . The "must be able…
I don't think people in this thread are insisting on a downloadable package. Just that the algorithm published in the write-up accurately describe the work that was actually done to achieve the stated results. If even that threshold can't be met, IMO the authors are polluting the literature with bad work.
Re: Differences between the word2vec paper and its implementation
#148Earlier quoted context omitted.
Is this really that common? That's disheartening, I want to spend time in academia but experiences like this are sucking the fun out for me...
I tried to make use of some public audio research and it was pretty bad. There was an audio comprehensibility competition a few years ago. Some of the papers submitted are still around, as well as the summary paper describing the results. But many papers are hard to find, and those that claimed to have source code available are hard to find --- i was able to get matlab sources for a few algorithms, but they somehow w…
this sounds interesting, would you care to reference the paper in question?
Re: Differences between the word2vec paper and its implementation
#149Earlier quoted context omitted.
There is a difference between unqualified people, and qualified but busy people. While there may be unqualified people reviewing papers, even qualified people who don't put in enough time and effort to review a paper comes to the same result. But the solution for that problem will look different. There's also a big difference between accepting a paper and judging if it is good . At a top conference like NIPS, lots of…
Submitting a paper to a conference for it to be published 11 months later is not going to work - the paper will be hopelessly obsolete by then. A better solution imo is to improve openreview.net process somehow, so that I'm motivated to go there, find papers relevant to my research (ideally get notified when such papers are posted for review), and leave a review (and perhaps vote on other reviews just like we do here…
This is ok if you can upload preprints on arxiv or equivalent. The obsolescence of papers already happens now, when a conference is 6 months after the submission, and the preprints system works fine.
Re: Differences between the word2vec paper and its implementation
#150Earlier quoted context omitted.
There is a difference between unqualified people, and qualified but busy people. While there may be unqualified people reviewing papers, even qualified people who don't put in enough time and effort to review a paper comes to the same result. But the solution for that problem will look different. There's also a big difference between accepting a paper and judging if it is good . At a top conference like NIPS, lots of…
Submitting a paper to a conference for it to be published 11 months later is not going to work - the paper will be hopelessly obsolete by then. A better solution imo is to improve openreview.net process somehow, so that I'm motivated to go there, find papers relevant to my research (ideally get notified when such papers are posted for review), and leave a review (and perhaps vote on other reviews just like we do here…
> Submitting a paper to a conference for it to be published 11 months later is not going to work - the paper will be hopelessly obsolete by then.
That's not how the VLDB process works. What they did was establish a PVLDB journal which has a monthly deadline, and it accepts 5-12 papers a month. The papers are public on the website a few months after acceptance. (See: https://vldb.org/pvldb/vol12.html) The VLDB conference is then all of the papers that appeared in PVLDB in the past year.
I would be in favor of exploring your model as well, but I also see the hybrid model developed by VLDB as superior to the standard conference submission and review process.