Too much of the research process is now shrouded by the opaque use of computers
1–10 of 55 posts
Re: Too much of the research process is now shrouded by the opaque use of computers
#2The solution is obvious - provide a github repository with all the code for the research. Provide clear ways to reproduce all the results of the research from raw data by running build.
Bonus points: Hirschware as software license.
Re: Too much of the research process is now shrouded by the opaque use of computers
#3The solution is obvious - provide a github repository with all the code for the research. Provide clear ways to reproduce all the results of the research from raw data by running build. Bonus points: Hirschware as software license.
Maybe even a container image / VM to ensure there's no subtle enviroment differences that alter the results. Even better with reproducible builds specifically so you can confirm it was created from the published source.
Re: Too much of the research process is now shrouded by the opaque use of computers
#4The solution is obvious - provide a github repository with all the code for the research. Provide clear ways to reproduce all the results of the research from raw data by running build. Bonus points: Hirschware as software license.
Isn't that what the article describes?
The core of the article is first recommendations on writing scripts to disambiguate cleanup tasks in data and organization/records management of files. It's then recommendations to use open source languages for scripting / analysis and for using git to maintain it all.
The trouble you run into is in the phrase "Provide clear ways to reproduce all the results."
There is "clear ways to," then there is "an exact method to." Academic life is cutthroat and research tenure and continued employment rely on showing that other peers are using their research. It's common practice to write an article in such a way that it looks "clear" but requires consultation with the researcher to understand precisely.
Ambiguity is part and parcel. The tough need here is core journals in each field being more rigorous on the sussing out of not-quite-good-enough methods statements to force researchers to be more open.
Re: Too much of the research process is now shrouded by the opaque use of computers
#5In my experience, scientists are like extremely junior developers with PhDs.
They name functions "wstok" instead of "whitespace_tokenizer" because it uses fewer keystrokes.
They can't be bothered to commit code. And when you do manage to get them to commit code, it is isn't the same code. It is close to the same code, but different. They think "close enough" is close enough, even though they recognize that without the exact code, it isn't the same thing. "Just a few tweaks will get you from that code to my code!"
This is a very difficult problem for a startup to solve.
EDIT 1 I read about other commenters writing about containers or just provide a github repository. The problem isn't the availability of the tools, the problem is convincing them to use the tools.
EDIT 2 Scientists don't give a damn about reading code, or what happens to the code the moment it leaves their hands. To quote one, "Why would I use a coding style? Why did you read the code? Code is not meant to be read!" This was a serious conversation.
EDIT 3 The most difficult thing is remembering they're not developers who care about developer tools. They're scientists who want to get results to their very exciting experiments.
EDIT 4 (as commented in reply to woah: we ("a startup") want to solve this problem because we employ scientists. See : https://news.ycombinator.com/item?id=10539433
EDIT 5 We've made great progress on working with scientists, and we're hoping to open source some of our tooling and write about what we've learned.
Re: Too much of the research process is now shrouded by the opaque use of computers
#6The solution is obvious - provide a github repository with all the code for the research. Provide clear ways to reproduce all the results of the research from raw data by running build. Bonus points: Hirschware as software license.
Maybe even a container image / VM to ensure there's no subtle enviroment differences that alter the results. Even better with reproducible builds specifically so you can confirm it was created from the published source.
IMO that's an overkill. It's sufficient that the result is reproduced once. If slightly different environment would produce enormously different results, that's interesting by itself.
The goal is that you can continue research based on previous results, not that you can reproduce exact results in an obtuse VM.
Re: Too much of the research process is now shrouded by the opaque use of computers
#7The solution is obvious - provide a github repository with all the code for the research. Provide clear ways to reproduce all the results of the research from raw data by running build. Bonus points: Hirschware as software license.
But if somebody would read the publication in, say, 20 years, then the whole platform on which the computation has to run might have become obsolete. Suddenly, you have to look for emulators. Or even for emulators running emulators.
Re: Too much of the research process is now shrouded by the opaque use of computers
#8This is exactly the problem that the Center for Open Science [https://cos.io/] is hoping to solve with the Open Science Framework [https://www.osf.io].
These are the same people behind the large Reproducibility Project that was on the top of HN not too long ago.
Re: Too much of the research process is now shrouded by the opaque use of computers
#9In some software-focused conferences, artifact evaluation has become a new, separate phase:
There is a separate committee charged with evaluating software (and other artifacts, like data sets), that come along with a paper.
This is a nice start to a process that could be adopted in non-CS fields for the evaluation of statistical results or software that analyzes data.
Re: Too much of the research process is now shrouded by the opaque use of computers
#10In my experience, scientists are like extremely junior developers with PhDs. They name functions "wstok" instead of "whitespace_tokenizer" because it uses fewer keystrokes. They can't be bothered to commit code. And when you do manage to get them to commit code, it is isn't the same code. It is close to the same code, but different. They think "close enough" is close enough, even though they recognize that without th…
I don't think that a startup can solve their coding problems. Only scientists can solve their own coding problems by having the diligence and attention to detail that one would expect of a high school dropout with 2 weeks of a coding bootcamp under their belt.