While it is definitely to the benefit of all that the bot emails authors when it finds mistakes, emailing when it doesn't find anything is a dark pattern. Reminds me of those bots that spam me after scraping my linkedin.
A bot crawled thousands of studies looking for simple math errors
161–170 of 221 posts
Re: A bot crawled thousands of studies looking for simple math errors
#162Earlier quoted context omitted.
> If you need to a 'chill' with the original researcher to get critical details, that's not science anymore. Certainly there is always the possibility that there was some details misunderstood, something that needs to be clarified, a print error, etc. Your "that's not science anymore" statement seems highly exaggerated. People are not supposed to communicate only via papers.
> People are not supposed to communicate only via papers. That we use written communication that persists through generations is the basis of science and society in general. If we cannot communicate sufficiently via papers, we're in a world of trouble.
I have a concrete counterexample. Let's say I write a paper presenting a model, plus some numerical results of large simulations. The code is based on gluing together various pieces of open source code. All these codes are typical scientist codes that are held together with duct tape. My paper is short, but I spent a lot of effort munging things together, and I'm fairly certain nobody can reproduce my results without my source code (preferably the whole environment) unless they spend a lot of time on trial and error like I did.
The tweaks I did to glue things together has no theoretical value and don't belong in the paper. As a practical matter, I can't fit a lot of source code into short paper format.
What do?
Re: A bot crawled thousands of studies looking for simple math errors
#163Earlier quoted context omitted.
Out of curiosity, do you remember what paper it was?
I'm not entirely certain but I think it was The modern synthesis of contrasting styles of thought in “east” and “west” *Nisbett, R. E., Masuda, T. (2003). Culture and point of view. Proceedings of the National Academy of Sciences, 100, September 16, 2003, 11163-11170
Interesting. As a physicist, I tend to put extra trust in PNAS credibility due to their history of critical editors and reviewers, and therefore am less likely to question results published there. Maybe it's the same fallacy, but at the same time we can't all be experts at everything and have to defer to trust at some point. Anyway, thanks for the contextual tidbit.
Re: A bot crawled thousands of studies looking for simple math errors
#164> and whether online critiques of past work constitute “bullying” or shaming. The PubPeer comments are just a tiny part of that debate. But it can hit a nerve. > Susan Fiske, a former president of the Association for Psychological Science, alluded to Statcheck in an interview with Business Insider, calling it a “gotcha algorithm.” > The draft of the article said these online critics were engaging in “methodological t…
[deleted]
Re: A bot crawled thousands of studies looking for simple math errors
#165Earlier quoted context omitted.
> If you need to a 'chill' with the original researcher to get critical details, that's not science anymore. Certainly there is always the possibility that there was some details misunderstood, something that needs to be clarified, a print error, etc. Your "that's not science anymore" statement seems highly exaggerated. People are not supposed to communicate only via papers.
The usefulness of a paper that doesn't stand on its own is rather limited, though.
Maybe a paper does stand on its own, with the large list of citations at the end. But, maybe some of those citations are journals that your institution doesn't subscribe to, or are historical and in another language, or et cetera.
There is a page limit to publications in high impact journals, and generally it's not great practice to utilize the limited space on the details of hurdles overcome.
I would argue that some of the most important papers in science don't really stand on their own... they need context and expertise that the paper can't and shouldn't cover.
Re: A bot crawled thousands of studies looking for simple math errors
#166Earlier quoted context omitted.
Please photograph and publish it. (Pics or it didn't happen?) This debate is important and shouldn't be muddied with hearsay on all sides. Thanks.
dmd is a respected HNer, so it's inappropriate to question him.
It is inappropriate to use that fact to dismiss a valid request for the evidence. Being famous in some (possibly popular) website does not mean that one's experience has not been biased.
Re: A bot crawled thousands of studies looking for simple math errors
#167Earlier quoted context omitted.
> People are not supposed to communicate only via papers. That we use written communication that persists through generations is the basis of science and society in general. If we cannot communicate sufficiently via papers, we're in a world of trouble.
I have a concrete counterexample. Let's say I write a paper presenting a model, plus some numerical results of large simulations. The code is based on gluing together various pieces of open source code. All these codes are typical scientist codes that are held together with duct tape. My paper is short, but I spent a lot of effort munging things together, and I'm fairly certain nobody can reproduce my results without…
Open source the code and supporting data.
Re: A bot crawled thousands of studies looking for simple math errors
#168Earlier quoted context omitted.
I have a concrete counterexample. Let's say I write a paper presenting a model, plus some numerical results of large simulations. The code is based on gluing together various pieces of open source code. All these codes are typical scientist codes that are held together with duct tape. My paper is short, but I spent a lot of effort munging things together, and I'm fairly certain nobody can reproduce my results without…
Open source the code and supporting data.
It's not that simple. What if some of it is proprietary? What if I'm not allowed to submit code because I need to be anonymous so reviewers can maintain impartiality? What happens when one of the upstreams update and breaks my code? Do I need to keep it updated? Forever?
Re: A bot crawled thousands of studies looking for simple math errors
#169Earlier quoted context omitted.
> new standards that didn't exist when the work was done Being correct is not a new standard. > This is simply not how things are done, in science or in society. Oh, I say! Positively indecent! A moral outrage! We can't have our morals compromised by scientific objectivity! > we do not and will never revisit all prior work and formally re-judge whenever standards or policies change. Have you ever heard of Principia M…
Please try to judge comments in their context and not look for excuses to try and humiliate someone with sarcasm. I was discussing standards with @int_19h, who brought up the issue of new methodologies. Of course being correct isn't a new standard, I agree with you, but that's not what was being discussed. Of course morals shouldn't be compromised by scientific objectivity. Again, you're arguing a straw man - that's…
My argument in context is that a paper cannot change because it has been published.
I think this statement is, if not completely untrue, grossly misrepresenting how existing papers are interacted with.
First of all papers, as with all publications, have errata published all the time. These errata may be included in future prints, or published in a separate location that can be looked up by people using the paper. Publishing errata is not a new occurrence, and although perhaps technically the original paper remains published unchanged, it is disingenuous to claim that this means the paper cannot change.
Modern publishing methods, such as the arXiv, allow for new versions of the paper to be uploaded, literally changing the published version of the paper.
As you point out yourself, literature reviews should point out issues with existing papers. Do you think that the original authors throw their hands in the air, thinking to themselves "oh well, it's published, nothing can be done"?? Of course not! If they are still engaged with the subject they either defend the paper, correct obvious mistakes, or continue experimentation or investigation in response.
To claim that errors should not be pointed out simply because the original authors can do nothing about the errors is diversionary at best. Of course errors in published results should be made public. How else can we trust any of the works?
If errors in existing research is always hidden, squelched, swept under the rug, we have no reason to trust it. It is the openness of research - publishing in the open, criticising in the open, discussing in the open - that allows us to trust research in the first place. Indeed, that trust is already eroded by revelations of systemic issues like p-hacking within published research.
You may be suggesting that posting these analyses to the individual papers was the wrong way to do it, that it would be better done in a literature review or paper.
I completely disagree.
It is essential that anyone looking to reference a paper with a glaring mistake in it (which many of those affected are) is able to see that mistake and correct for it. Leaving the old research be is just ensuring that incorrect ideas are allowed to propagate, and have more of an impact than they ever should.
Re: A bot crawled thousands of studies looking for simple math errors
#170Earlier quoted context omitted.
dmd is a respected HNer, so it's inappropriate to question him.
It is inappropriate to use that fact to dismiss a valid request for the evidence. Being famous in some (possibly popular) website does not mean that one's experience has not been biased.
Go five levels up from your comment and you might find a joke you're missing—unless you're in on it too and it's sailing right over my head