Differences between the word2vec paper and its implementation

I am one of those people who actually did the extra weeks/months to properly test/review/document/release my code and data sets (you can apt-get install my "research artifacts").

In retrospect, it was a poor use of my time and a poor use of my sponsoring institution's time. "apt-get install" is NOT what we mean by "reproducible" in science.

High-quality or easy to install code is not necessary for a result to be reproducible. True reproduction would mean coding the algorithm from scratch by following the written description, and that's what it literally means in most other fields of science.

You can't download & install a Large Hadron Collider in an afternoon. Does that mean the LHC experiments are "not reproducible"? Of course not.

But that's not even the important point. The really important point is that, in most cases, high-quality code is not even sufficient for a result to be reproducible! See: the article we're discussing.

IMO, the "blindly rerun the code" definition of reproduction is actually a HUGE barrier to creating a true culture of reproducability in computer science. It results in super lazy reviewing where "public source code that's easy to install and puts the correct-sounding shit into STDOUT" becomes a stand-in for "paper actually describes a novel idea in enough detail that it can be truly reproduced".

> So what is the optimal outcome here?

An optimal allocation of scientists' time and effort.

As a scientist who has actually done that leg work, I don't think packaging code so that it runs with a single click is the best use of public money in science in 99.9999% of cases. That time is much better spent on writing and other dissemination explaining the ideas that make the code work (in some cases well-documented source code is the best description but in other cases prose is much more effective and illuminating). Or on coming up with new ideas that are even better than the old ones.

Which I guess is just another way of saying that scientists should spend their time on science, not engineering.

P.S. When shitting on "scientists" for not being good enough software engineers, please remember who's going to be doing the actual work you're demanding. It's mostly phd students who make $30K/yr. And they have to do this work in their free time because their 60 hr/wk day job is fully allocated to doing the actual science. I.e., treat scientists who maintain their code as you would treat FOSS contributors who are making 5x-10x+ less than you while working longer hours. Because maintaining high-quality code is something they are almost certainly doing in their free time.

A very large number of research papers I've read in my life were so poorly written as to being near impossible to understand.

The time it took to infer the actual meaning of certain parts of certain papers was vastly larger than just reading the code would have been.

It has become very clear to me that being a strong CS researcher does not necessarily imply mastery of language or even the very basic ability to explain one's ideas. As a matter of fact, with very few exception (eg Feynmann), the two skills seem to be at odds with one another.

On the other hand, code can be read and understood much more readily, especially when it can be experimented with, if only to insert printfs in it to understand what it does.

The culture of scientific publication in CS must change. The "I'm pressured by my advisor and therefore have no time to publish my code" argument is complete bollocks.

The code should come first and your advisor should pressure you to publish that first (assuming it actually works).

If someone wants to read your poorly worded explanation of how it works, fine, publish a paper.

Show me the code first.

Welcome to the planet, enjoy your stay!

I'll take you at your word.

You want to just drop in and drop bombs and pretend that the information you are presenting is in some sense useful. Other folks pointed out that you are wrong.

But you also want the reputation. You want the respect for revising your opinion.

You're you from your perspective, but from everyone else's perspective you're not you, you're the basest troll that has ever posted on HN, because you have no rep, no history, not provenance.

It's funny to me, because you want it both ways. It's pretty hilarious.

I shouldn't feed the trolls. I know better. Enjoy being a complete waste of time to everyone that glances at your writing!