[deleted]
Fake scientific papers are alarmingly common
51–60 of 105 posts
Re: Fake scientific papers are alarmingly common
#52Unfortunately, it's an open secret that fake or low-effort almost useless papers are very common in every area of scientific research. Typically, it doesn't affect people working in that specific area - they develop/have a sixth sense to detect bullshit papers - it comes with experience but depends on several factors including the authors reputation, their institution (for the first screening), what journal/conferenc…
My favourite example is the grievance studies ( https://en.wikipedia.org/wiki/Grievance_studies_affair ). The authors published, among others, portions of directly copied Mein Kampf. In one "study" submitted, they claimed to have observed thousands of hours of dogs having sex in parks, to observe the "patriarchal" linked to "rape culture". The entire thing is a horrible indictment on the level of scrutiny undertaken…
I have no sympathy for social science journals, but when you look at the details of this study, it's way less obvious that the rumor says it is. On the tens of articles they have proposed, the majority was rejected. The article "copied from Mein Kampf" was taking sentence from the book but changing words to create sentences that were scientifically correct (for example: "this social class is bad and we should avoid it" into "stress is bad and we should avoid it"), which means that the article content in itself had no reason to be refused.
It's very ironical that this study that was all about "bad science" since then created a totally whimsical rumor on the real situation.
Re: Fake scientific papers are alarmingly common
#53Sadly, this doesn't even include the studies with authors who produce poor experiments and theories, or go out of their way to prove their results; effectively, generating additional scientific publication waste we have to sift through to find genuine material, or worse off, that people then use to create policies impacting large populations that are doomed to fail in the long-run. The image this creates for me, is b…
In essence, there's more than enough to deal with regarding bad science that was done in good faith. There have got to be better ways to filter out bad science offered in bad faith.
I think the incentive structure has to be fixed - either making paper mills unnatractive, or removing the demand for their services, whichever.
I think filtering out bad faith efforts is too challenging because the pool of people capable of doing so is so limited, hell it might take longer to review and reject such a thing than to make it.
Re: Fake scientific papers are alarmingly common
#54I found one the other day in the area of finance. The Chinese researchers claimed to have discovered a small tweak to a long established indicator which they described as giving a remarkable increase in r-squared value across a cross section of markets.
Sounds great, who wouldn't want to use this? So I implemented and find that their increase was due entirely to applying a log transform of the input variables. The resulting clusters were tighter, but it had zero predictive capability.
Very disappointing but in my experience, this is not uncommon.
Re: Fake scientific papers are alarmingly common
#55Why Most Published Research Findings Are False - https://journals.plos.org/plosmedicine/article/file?id=10.13...
Re: Fake scientific papers are alarmingly common
#56> Such manuscripts threaten to corrupt the scientific literature, misleading readers and potentially distorting systematic reviews. Is treating "the scientific literature" as a single thing perhaps a habit worth giving up? As convenient as it would be to be able to just blindly trust something because of where it is published, that model hasn't shown itself to be especially robust in other cases (e.g. the news media)…
I disagreed with you until the last paragraph. Lots of things authentically just rely on a high degree of trust and I suspect trying to engineer human systems to be zero trust will make them deeply pathological.
But tempering our expectations while working to meaningfully improve on conditions? Aces, all for it.
Re: Fake scientific papers are alarmingly common
#57Earlier quoted context omitted.
No. They use 400 known fakes and 400 matched (presumed) non-fakes to estimate the sensitivity and specificity of their indicator, then apply that indicator to the full universe, then employ the estimated sensitivity and specificity to the obtained measurement to estimate the approximate actual rate of false papers. If you know the true prevalence of a disease in a population, and the sensitivity and specificity of yo…
> Furthermore, they’re explicitly saying that “red flagging” by their simple indicator doesn’t mean that the paper is fake, but that it merits higher scrutiny. Then they and science should change their sensationalist headline. It's ironic that a paper about fakeness of something uses a borderline misleading title.
You’re not wrong, but it is everyone’s own responsibility to read the article and not just the headline.
Re: Fake scientific papers are alarmingly common
#58If people think that 100%-fake papers, with completely made up data and process are bad... wait until people learn how bad 30%-fake papers are, with real cherry-picked data and absurd levels of p-hacking :p
Re: Fake scientific papers are alarmingly common
#59'In an example of Brandolini's law [...] "It took this guy 15 minutes to make his video and it took me three days to fact-check."' [1] [1] https://en.m.wikipedia.org/wiki/Brandolini%27s_law
[deleted]
Re: Fake scientific papers are alarmingly common
#60> Such manuscripts threaten to corrupt the scientific literature, misleading readers and potentially distorting systematic reviews. Is treating "the scientific literature" as a single thing perhaps a habit worth giving up? As convenient as it would be to be able to just blindly trust something because of where it is published, that model hasn't shown itself to be especially robust in other cases (e.g. the news media)…
I disagreed with you until the last paragraph. Lots of things authentically just rely on a high degree of trust and I suspect trying to engineer human systems to be zero trust will make them deeply pathological. But tempering our expectations while working to meaningfully improve on conditions? Aces, all for it.
I agree that zero trust is in most cases a problematic goal. It's really root-of-trust vs web-of-trust that I'm on about here.
If peer review is the product then the trust should be peer to peer. It feels like we're treating the publishers themselves as an authority, which I dislike.