Earlier quoted context omitted.
Industry doesn't need peer review. The proof of the pudding is in the sales.
Then why does it dress its marketing up in academic paper format?
In want of sales.
Replace peer review with “peer replication” (2021)
271–280 of 351 posts
Re: Replace peer review with “peer replication” (2021)
#272Earlier quoted context omitted.
It’s also very easy to come up with bad ideas — I did plenty of that and I still do, albeit less than I used to. Finding an idea that is novel, interesting, and tractable given your time, skills, resources, and knowledge of the literature is hard, and maybe the most important skill you develop as a researcher. For a reductive example, the idea to solve P vs NP is a great one, but I’m not going to do that any time soo…
I'd say it is a bad idea because you're not going to succeed at it.
Sure because that’s an easy example and kind of my point. When you’re starting your PhD, it can be hard to determine what questions you can and cannot answer given your skills and surrounding literature. That comes from experience and trying (and failing) a lot.
Re: Replace peer review with “peer replication” (2021)
#273Earlier quoted context omitted.
This is the final stage of capitalism, where market success is conflated with scientific rigor.
Engineering, not scientific. High Sales = A large number of people can attest to the engineered good or service being of high enough quality that they will exchange hard earned money for the ability to use it. Peer Review = Some folks who derive self-worth from citations ask you to add a citation to their work and you do it because you'll probably need to ask them for the same some time later.
This doesn't really account for fraud or abuse. Plenty of profitable companies whose product is a scam, or has other factors that make it successful.
Market success isn't the same thing as validity.
Re: Replace peer review with “peer replication” (2021)
#274Earlier quoted context omitted.
It's the general public that equates "peer reviewed" with "definitely correct, does not need to be questioned".
There is genuine merit in peer review though. A lot of the marketing "papers" from OpenAI and Google would benefit from going through a peer review process instead of scholarwashing and uploading straight to Arxiv. One simple example, (from memory) the Bard paper doesn't include results for experiments in which GPT-4 outperforms it. As a result, people come away from these works with an inflated understanding of thei…
You can often see a p-hacked study from a mile away because they measure large numbers of unnecessary variables. They then pick those 4 variables that yield a probably false signal and publish on it. One would think these would never pass peer review, but they're regularly peer reviewed, published, and then, shockingly enough, fail to replicate. Hypothesizing after the results are known falls in the same bucket here. This is why pre-publishing kicked off, yet it's also hardly a savior.
The point I make is that peer review can not be guaranteed to 'fix' science in any way we might like. The Sokal Affair [1] has now been replicated repeatedly, including in peer reviewed journals. The most recent one even got quite cheeky and published it under the names "Sage Owens, Kal Avers-Lynde III" - Sokal III. [2] It always preys on the same weakness - bias confirmation.
[1] - https://en.wikipedia.org/wiki/Sokal_affair
[2] - https://www.nationalreview.com/news/academic-journal-publish...
Re: Replace peer review with “peer replication” (2021)
#275Back when I was active in academia, our publishers were reluctant to print source code or even repository links (that was largely before GitHub) but they could still share a paper source on demand. If you reference someone else's paper and want to quote some formula, it is easier and less error prone to copy rather than retype.
At that point I thought about making a TeX interpreter so one could easily "run a paper" on their own data to see if the papers claims hold. As it turned out, people often write the same formula in multiple ways and to make a TeX interpreter you'd have to specify a "runnable" subset and convince anyone to use that subset instead of what they got used to. So the idea stalled.
In a few years, publishing a GitHub link along the paper became the norm, and the problem disappeared. At least in applied geometry, people do replicate each other results all the time.
Re: Replace peer review with “peer replication” (2021)
#276I don't see how this could ever work, and non-scientists seem to often dramatically underestimate the amount of work it would be to replicate every published paper. This of course depends a lot on the specific field, but it can easily be months of effort to replicate a paper. You save some time compared to the original as you don't have to repeat the dead ends and you might receive some samples and can skip parts of…
Please excuse my ignorance, but I'm not convinced.
What are we supposed to do in a hundred years when the scientists of today are dead and we have a bunch of results with important implications that aren't documented well enough to replicate?
Re: Replace peer review with “peer replication” (2021)
#277For a while Reddit had the mantra “pics or it didn’t happen”. At least in CS/ML there needs to be a “code or it didn’t happen”. Why? Papers are ambiguous. Even if they have mathematical formulas, not all components are defined. Peer replication in these fields is an easy low hanging fruit that could set an example for other fields of science.
CS and ML are my field, although I'm no longer active in research. I always made a code archive available. Want to replicate? Download and run.
This should be standard now, in the age of GitHub, GitLab, et al. If a paper discusses an implementation, but doesn't provide code, it is probably BS.
Re: Replace peer review with “peer replication” (2021)
#278Earlier quoted context omitted.
Academia's values are not objective. Why is it that replicating or refuting a study is not seen on par as being a co-author of said study? There is nothing set in stone preventing this, only the current academic culture.
Because I want to do original research, and be known for doing original research. Only if I fail at that, I might settle for being a guy who reproduces others’ work (which basically means the transition from a researcher to an engineer).
So you want to be known for original results, which cannot be confirmed to be true? So what kind of results are they?
Also your mentality is exactly part of the problem: you arrogantly believe that replication work is beneath you and that originality is all that matters.
Re: Replace peer review with “peer replication” (2021)
#279Earlier quoted context omitted.
It's the general public that equates "peer reviewed" with "definitely correct, does not need to be questioned".
There is genuine merit in peer review though. A lot of the marketing "papers" from OpenAI and Google would benefit from going through a peer review process instead of scholarwashing and uploading straight to Arxiv. One simple example, (from memory) the Bard paper doesn't include results for experiments in which GPT-4 outperforms it. As a result, people come away from these works with an inflated understanding of thei…
What is honestly a lot more compelling than peer review is multiple supporting pieces of evidence from different sources. One result might be spurious, but if you can find a couple independent studies showing as much its probably a real phenomenon.
Re: Replace peer review with “peer replication” (2021)
#280Earlier quoted context omitted.
I think you don't understand how much work is involved in just building the techniques and expertise to pull some experiments off (let's not even talk about the equipment). Even if someone meticulously documents their process, it could still take months to replicate the results. I'm familiar with lithography/nanofabrication and I know that it is typically the case that a process developed in one clean-room can not be…
Months. Haha. I previously worked in agricultural research (in the private sector), and we spent YEARS trying to replicate some published research from overseas. And that was research that had previously been successfully replicated, and we even flew in the original scientists and borrowed a number of their PhD students for several months, year after year, to help us try to make it work. We never did get it to fully…
> We never did figure out why it worked in one part of the world but not another
Doesn't that indicate further research is needed? It sounds fascinating to me. (I know it isn't interesting for the people who couldn't get it working.) It also might indicate that the original research was incomplete in the sense that it might be a fluke due to specific conditions in the original country which isn't universal.