91–100 of 351 posts
At least in CS/ML there needs to be a “code or it didn’t happen”. Why? Papers are ambiguous. Even if they have mathematical formulas, not all components are defined.
Peer replication in these fields is an easy low hanging fruit that could set an example for other fields of science.
High impact journals [6] tend to prefer exciting, novel, and positive results (we tried new thing and it worked so well!) vs negative results (we mixed up a bunch of crystals and absolutely none of them are room-temp superconductors! we're sure of it!).
The result is that cherry picking data pays, leaning into confirmation bias pays, publishing replication studies and rigorous but negative results is not a good use of your academic inertia.
I think that creating a new category of rigor (i.e. journals that only publish independently replicated results) is not a bad idea, but: who's gonna pay for that? If the incentive is you get your name on the paper, doesn't that incentivize coming up with a positive result? How do you incentivize negative replications? What if there is only one gigantic machine anywhere that can find those results (LHC, icecube, etc, a very expensive spaceship)?
There might be easier and cheaper pathways to reducing bad papers - incentivizing the publishing of negative results and replication studies separately, paying reviewers for their time, coming up with new metrics for researchers that prioritize different kinds of activity (currently "how much you're cited" and "number of papers*journal impact" things are common, maybe a "how many results got replicated" score would be cool to roll into "do you get tenure"? See [3] for more details). PLoS publish.
I really like OP's other article about a hypothetical "Journal of One Try" (JOOT) [2] to enable publishing of not-very-rigorous-but-maybe-useful-to-somebody results. If you go back and read OLD OLD editions of Philosophical Transactions (which goes back to the 1600's!! great time, highly recommend [4], in many ways the archetype for all academic journals), there are a ton of wacky submissions that are just little observations, small experiments, and I think something like that (JOOT let's say) tuned up for the modern era would, if nothing else, make science more fun. Here's a great one about reports of "Shining Beef" (literally beef that is glowing I guess?) enjoy [5]
[1] https://www.ncbi.nlm.nih.gov/pmc/articles/PMC6668985/ [2] https://web.archive.org/web/20220924222624/https://blog.ever... [3] https://www.altmetric.com/ [4] https://www.jstor.org/journal/philtran1665167 [5] https://www.jstor.org/stable/101710 [6] https://en.wikipedia.org/wiki/Impact_factor, see also https://clarivate.com/
Earlier quoted context omitted.
Your analysis seems to portray all scientists as pure hearted. May I remind you of the latest Stanford scandal where the president of Stanford was found to have manipulated data? Today, publications do not serve the same purpose as they did before the internet. It is trivial today to write a convincing paper without research and getting that published(www.theatlantic.com/ideas/archive/2018/10/new-sokal-hoax/572212/&s…
No subset of humanity is “pure hearted.” Fraud and malice will exist in everything people do. Fortunately these fraudulent incidents seem relatively rare, when one compares the number of reported incidents to the number of publications and scientists. But this doesn’t change anything. The benefit of scientific publication is to make it easier to detect and verify incorrect results , which is exactly what happened in…
In this case, we are not even aware of most events of fraud/"bad papers"/manipulation - the "crisis" is that we are losing faith in the science we are doing - results that were cornerstones of entire fields are found to be nonreproducible, making all the work built on top of it pointless.(psychology, cancer, economics, etc - I'm being very broad)
At this point, we don't know how deep the rot goes. We are at the point of recognizing that it's real, and looking for solutions. For car accidents, we're past that - we're just arguing about what are the best solutions. For the replication crisis, we're trying to find a way forward.
Like that scene in The Thing, where they test the blood? We're at the point where we don't know who to trust.
Ps: what's a tfa?
I like the idea of splitting "peer review" into two, and then having a citation threshold standard where a field agrees that a paper should be replicated after a certain number of citations. And journals should have a dedicated section for attempted replications. 1. Rebrand peer review as a "readability review" which is what reviewers tend to focus on today. 2. A "replicability statement", a separately published docu…
Every experimental paper I've ever read has contained an "Experimental" section, where they provide the details on how they did it. Those sections tend to be general enough, albeit concise. In some fields, aside from specialized knowledge, good experimental work requires what we call "hands." For instance, handling air sensitive compounds, or anything in a condensed or crystalline state. In my thesis experiment, some…
Peer review does not serve to assure replication, but assure readability and comprehensibility of the paper. Given that some experiments cost billions to conduct, it is impossible to implement "Peer Replication" for all papers. What could be done is to add metadata about papers that were replicated.
Isn't readability and comprehensibility the job of the editor/journal to check. (after all they're actually paid) maybe not for conference, but peer review is more for checking if the methodology, scope, claim, direction, conclusion and relevances is sound&trustable. At least that's my understanding
Also, most work goes to conferences; journals typically publish longer versions of published works.
Earlier quoted context omitted.
They're basically no barriers to publication. There are a number of normal journals that publish everything submitted if it appears to be honest research.
Not nice journals, though. At least not in my experience but that’s probably very field-dependent. It’s not uncommon to get a summary rejection letter for lack of novelty and that is one aspect they stress when they ask us to review articles.
It's basically a catch 22. We want replication in prestigious journals, but any Journal with replications becomes less novel and prestigious.
It all comes down to what people value about journals. If people valued replication more than novelty, replication journals would be the prestigious ones.
It all comes back to the fact that doing novel science is considered more prestigious than replication. Institutions can play all kinds of games to try to make it harder for readers to tell novelty apart from replication, but people will just find new ways to signal and determine the difference.
Let's say we pass a law that prestigious journals must published 50% replications. The Prestige from publishing in that journal will just shift to publishing in that journal with something like first demonstration in the title or publishing in that journal Plus having a high citation or impact value.
It is really difficult to come up with the system or institution level solution when novelty is still what individuals value.
As long as companies and universities value innovation, figure out ways to determine which scientists are innovative, and value them more
I spent a lot of my graduate years in CS implementing the details of papers only to learn that, time and time again, the paper failed to mention all the short comings and fail cases of the techniques. There are great exceptions to this. Due to the pressure of "publish or die" there is very little honesty in research. Fortunately there are some who are transparent with their work. But for the most part, science is dro…
I don't see how this could ever work, and non-scientists seem to often dramatically underestimate the amount of work it would be to replicate every published paper. This of course depends a lot on the specific field, but it can easily be months of effort to replicate a paper. You save some time compared to the original as you don't have to repeat the dead ends and you might receive some samples and can skip parts of…
> I don't see how this could ever work, and non-scientists seem to often dramatically underestimate the amount of work it would be to replicate every published paper. They also tend to over-estimate the effect of peer review (often equating peer review with validity). > If someone cares enough about the work to build on it, they will replicate it anyway. And in that case they have a good incentive to spend the effort…
I have gone down the rabbit hole of engineering research before and 90% of the time I’ve managed to find an anecdote or subsequent research footnotes or actual subsequent research publications, that substantially invalidated the lofty claims of the engineers in the 70s or 80s (which is amazing still despite this, a genuine treasure trove of research unused and sometimes useful aerospace engineering research and development) and unfortunately outside the few proper publications, a lot of the invalidations are not properly reverse cited research material and I could have spent a week cross referencing before I spot the link and realise the unnamed work they are saying they are proving wrong is actually some footnotes containing the only published data (before their new paper) on some old work that has a bad scan copy on the NASA NTRS server under some obscure title and no related keywords to the topic the research is notionally about…
Academic research can genuinely suck sometimes… particularly when you want to actually apply it.
How do you replicate a literature review? Theoretical physics? A neuro case? Research that relies upon natural experiments? There are many types of research. Not all of them lend themselves to replication, but they can still contribute to our body of knowledge. Peer review is helpful in each of these instances. Science is a process. Peer review isn't perfect. Replication is important. But it doesn't seem like the aut…
The purpose of science publications is to share new results with other scientists, so others can build on or verify the correctness of the work. There has always been an element of “receiving credit” to this, but the communication aspect is what actually matters from the perspective of maximizing scientific progress. In the distant past, publication was an informal process that mostly involved mailing around letters,…
I’ll pile on to say that you also have the variable of how the non-scientist public gleans information from the academics. Academia used to be a more insular cadre of people seeking knowledge for its own sake, so this was less relevant. What’s new here is that our society has fixated on the idea that matters of state and administration should be significantly guided by the results and opinions of academia. Our enthusiasm for science-guided policy is a triple whammy, because 1. Knowing that the results of your study have the potential to affect policy creates incentives that may change how the underlying science is performed 2. Knowing that results of academia have outside influence may change WHICH science is performed, and draw in less-than-impartial actors to perform it 3. The outsized potential impact invites the uninformed public to peer into the world of academia and draw half-baked conclusions from results that are still preliminary or unreplicated. Relatively narrow or specious studies can gain a lot of undue traction if their conclusions appear, to the untrained eye, to provide a good bat to hit your opponent with.