Similar things can be said about the textbook and the lecture
The Scientific Paper is Obsolete (2018)
81–90 of 175 posts
Re: The Scientific Paper is Obsolete (2018)
#82I feel like the website paperswithcode.com addresses this very well, especially with their feature "quick start in Colab". For example, here's the top paper on the website as of now: https://paperswithcode.com/paper/towards-real-world-blind-fa... . Instead of going through the process of cloning a repo, initializing a fresh Anaconda environment from scratch, reading through nebulous, haphazard documentation about how…
It's really disappointing that technical societies like the ACM and IEEE haven't done this already. For many journals and conferences there isn't even a way to submit the code or other digital artifacts with the PDF. A few have badging for whether digital artifacts are provided and whether the results have been reproduced or repeated by others - steps in the right direction at least. As much as I intensely dislike th…
> ...For many journals and conferences there isn't even a way to submit the code or other digital artifacts with the PDF.
Why not to stuff it into a repo and reference it in the text?
Of course, a central location within a stable institution helps with continuity of availability of such repo. But at least this gives you some control over it.
Re: The Scientific Paper is Obsolete (2018)
#83Earlier quoted context omitted.
> The core research curriculum of nearly every scientific field I’ve seen, STEM or otherwise, is that the data needed for replication are non-negotiable. A paper that doesn’t include it would be table rejected by any editor. Or one would hope. This is taught at the UNDERgraduate level, for heaven’s sake. This may vary based on discipline, but in both the subdisciplines of experimental and theoretical physics I was in…
Serious question: if the experiment can't be reproduced to be verified, how does the paper provide more proof than a blanket 'trust me'? From someone who isn't an academic, isn't this letting politics come before science?
> if the experiment can't be reproduced to be verified, how does the paper provide more proof than a blanket 'trust me'?
It doesn't, and it is a big "trust me". People review papers based on the merits of the idea and methodology, and then tend to trust the results. Of course, if the results are very "significant" (e.g. cold fusion), then it will be scrutinized more, people will fail to reproduce, and they will harass the author. 99% of papers don't fall in this category, though.
> isn't this letting politics come before science?
Yep. The games at play are often: "How do I write my paper in the most convincing way?" and "As a referee, this paper is hurting the research work I am currently doing. What is the best way to reject this paper?"
The extremely annoying part was I felt I was back to taking literature courses, where I'm graded on very subjective metrics. It was horrible, especially when all my work was extremely objective. However, the publishing system is not incentivized to be that objective.
Simple example: A colleague's paper was rejected because he explained a phenomenon using method A, and the referee complained there was no mention of method B. Method B was the hot topic of the day. Neither method A nor method B had good empirical data to support it - it was almost purely theoretical at that point. But that community was gravitating towards method B, and really did not want to see alternative explanations.
Re: The Scientific Paper is Obsolete (2018)
#84I think this is a great idea theoretically, but in reality for most papers I don't want to see the data/underlying code. While it would be great to publish data/code with the paper (in the field I've worked on the most, astronomy, most data is already published with the paper anyways), I don't want/need to look through a notebook with the underlying code of the paper in order to just read the intro/conclusions (and m…
You should always want to have the underlying code available. Without the exact procedures they used to process their data, the only kind of "using their conclusions" you can do is the superficial "take it at face value" kind. So many important details get hand-waved away in papers that say things like "we used the well known blahblahblah method to analyze the data." If you do it right, the code should in no way inte…
[deleted]
Re: The Scientific Paper is Obsolete (2018)
#85Earlier quoted context omitted.
Tiktok videos are primarily used for entertainment, rather unlike Jupyter notebooks and source code repositories. Surely you have a more serious objection.
You don't read git repos for entertainment?!
For entertainment, I tend to read about things outside my field — things like In the Pipeline where I learn about FOOF and chlorine triflouride and freaky molecules like jawsamycin (insert shark theme here). I also watch Chemical Safety and Hazard Investigation Board videos on YouTube, like that time a refrigerator accident at a poultry plant caused hydraulic shock and released a massive cloud of ammonia, and ~150 contractors hanging out across the river working on Deepwater Horizon cleanup measures got sent to the hospital.
Re: The Scientific Paper is Obsolete (2018)
#86Earlier quoted context omitted.
Yes but chances are it only appears to work because the analysis code has bugs. So first i want to check the code and that it works before i put effort into understanding the idea.
So let me clarify. You're saying that the majority of CS papers only appear to work because the analysis code has bugs? And that checking the code (presumably also the analysis code) is easier than "understanding the idea"? Neither of those ring true to me, but your mileage may vary.
> CS papers
I don't think any specific field was mentioned
Re: The Scientific Paper is Obsolete (2018)
#87Earlier quoted context omitted.
The whole point of the field of statistics is that you can carry out statistical tests and analysis on a sample; you don’t need all of the data.
It doesn’t really work like that. For instance, imagine you have a simulation with billions of particles in it. To construct reduced data you may need to use many fields (position, temperature, composition) of all particles over many outputs (usually at different times).
In that case you shouldn’t need to ship the data at all. Just include the code for the simulation and let the rescuers run it to generate the data themselves.
Re: The Scientific Paper is Obsolete (2018)
#88I think this is a great idea theoretically, but in reality for most papers I don't want to see the data/underlying code. While it would be great to publish data/code with the paper (in the field I've worked on the most, astronomy, most data is already published with the paper anyways), I don't want/need to look through a notebook with the underlying code of the paper in order to just read the intro/conclusions (and m…
I have made an experiment with my last paper: Write everything from scatch in Jupyter Notebook, including data preprocessing and generation of all figures (etc.) (10 Notebooks in total).
Start of the conceptualization was in 2017, we just submitted it 2 weeks ago (it got desk rejected for not fitting the journals topic).
I learned a lot and it was definitly worth it. The next paper will be easier with this knowledge. Nonetheless, there is an overhead and I feel that this overhead is not valued with the current makeup of journals, where you really need to dig deep to find any supplementary materials.
Re: The Scientific Paper is Obsolete (2018)
#89As a practicing scientist, I firmly believe the world would be much better off if we simply published version-controlled Jupyter notebooks on a free site, such as GitHub or ArXiv.
Do you do so? If not, why not?
I'm not an academic, but a physicist working in R&D at a company. So my "papers" are only for internal consumption, and not earth shattering anyway. My colleagues and I are using Jupyter extensively.
My observation, from seeing papers that have been written in Jupyter, and observing how people work, is that Jupyter will first gain traction in disciplines that are already computation-heavy, and where open software is closer to the front end of the data pipeline.
For instance in my case, I develop measurement instruments, so everything I make is computerized, by me or my colleagues. While "raw data" may be in the form of things like voltages, they are almost immediately turned into a Python friendly data format by code that I wrote myself. So I'm up to my armpits in data and code just to get my experiments even barely working in the first place. I have a computer with coding tools literally at every bench in the lab. Jupyter is my lab notebook, and often my "report" is just the same notebook, dressed up with some readable commentary.
Now, contrast that with somebody like a synthetic bench chemist. The data that they get may be in computer readable form, but they rarely do any coding during the course of a project. For analysis, they're satisfied with the computations rolled into their instrument software, or Excel. And a fair amount of their analysis is in the form of explaining their way through an argument that connects data from disparate measurement techniques, using pictures and graphs. They don't program. The ones who can program have gone into software development. The ones who are using Jupyter are motivated to use it, as an end unto itself. Bringing that stuff together in Jupyter wouldn't help much. Many of their journals do require submission of raw data.
This is similar to questions about why so many people use Excel. I think you have to actually immerse yourself in the specific work environment an observe or even experience what people are experiencing, what they're actually studying, how they think, and so forth. There's a certain Chesterton's Fence aspect to discussions that start with the premise that some widespread activity is hopelessly broken beyond repair and must be immediately abolished.
Re: The Scientific Paper is Obsolete (2018)
#90Earlier quoted context omitted.
I think this almost every time I read the paper. It’s like Linus’ “show me the code.” I just want papers now to “show me the data and the code.” And include a discussion about why these results are important. I think it’s a great time for the scientific community to improve transparency on these fronts. Sincerely, someone who reads a lot of research but contributes none because I’m an amateur. Edit: when I say data,…
> I just want papers now to “show me the data and the code.” But the code is secondary to the idea. The idea and the discussion around how it was arrived at and what it means is the key thing. The code is just there to implement it. You could code the same idea ten different ways.
I translated what you said in the context of mathematics:
“But the proof is secondary to theorem. The theorem and the discussion around how it was arrived at and what it means is the key thing. The proof is just there to show it’s true. You could write a proof for the same theorem ten different ways.”
Which is all true. But man it wastes so much time having to re-prove everything. Also some lemmas/theorems are so hard to prove. It’s much easier when you see some incredible statement and can’t believe it’s true to look at the proof and see where the mistake / contentious part is.