Live data from Hacker News

A bot crawled thousands of studies looking for simple math errors

vox.com

81–90 of 221 posts

Re: A bot crawled thousands of studies looking for simple math errors

#81
post #61

Earlier quoted context omitted.

There are two options here: Use a bot to find tons of mistakes automatically and risk coming across as rude. Or Let social trivialities dominate scientific discourse and let most of those mistakes go unchecked forever because there's no feasible way to "politely" address hundreds of authors who made mistakes and keep checking back to make sure they actually fixed the mistake. The former is clearly the preferable choi…

Science being a social activity, I think you have to justify the presumption that we are talking about social trivialities here.

>Science being a social activity

Hoo boy. Research might be a social activity, but science is the application of probabilistic reasoning to evidence collection. Science isn't a "social activity" any more than topology is. Any social complications are entirely incidental.

Re: A bot crawled thousands of studies looking for simple math errors

#82
post #66
post #61

Earlier quoted context omitted.

There are two options here: Use a bot to find tons of mistakes automatically and risk coming across as rude. Or Let social trivialities dominate scientific discourse and let most of those mistakes go unchecked forever because there's no feasible way to "politely" address hundreds of authors who made mistakes and keep checking back to make sure they actually fixed the mistake. The former is clearly the preferable choi…

There is a third option that's better than either one you proposed. Get people writing new papers to use statcheck before they publish. The only rudeness here was an avoidable choice - publishing statcheck's results on a huge set of already published papers. The statcheck authors chose to do that for exposure, they even said so, it was not for posterity or the scientific well being of the community. I don't personall…

And what happens when a new fact-checking algorithm comes out? What I said in my comment.

Re: A bot crawled thousands of studies looking for simple math errors

#83
post #59

Earlier quoted context omitted.

> How exactly could you possibly have any sort of understanding for that? I'm also confused by your question, so I can relate to you. :) Despite your warning, the way you phrased that question does sound like an attack, it implies that you know my experience does not allow me to speak on this subject, which you do not know. Why is it hard to think that I can empathize? What do you think you know about me that makes i…

> it implies that you know my experience does not allow me to speak on this subject Not in the least my intention. I could not understand how anyone, regardless of knowledge level, could empathize with the quote as stated. From your response, it seems then that your quote wasn't meant in the way it seemed to me. It's not even the unsolicited nature of the report, but any error that causes irritation, and while possib…

Yes having any error pointed out can be frustrating.

However, the unsolicited nature of it may play an important role in this case. In order to empathize with the feelings of people who might have been irritated it is helpful to understand the academic paper publishing process.

The authors all went through a stressful process of submitting their hard work to a journal and then being evaluated by a panel of "experts". Many of them had to make changes to their papers and resubmit them in order to get published.

There's a level of understanding and expectation about how this process works. The papers aren't normally open for public comment before publication, and they don't normally get public comment after publication. They're evaluated by people in the field, and presented at conferences, and then referenced in other papers if they're influential.

Having a unknown third party with brand new possibly buggy software cast public aspersions on a paper after the fact, at a time when nothing can be done about it, is simply not helpful to the authors and is not how reviews normally happen. It's very easy to see why authors wouldn't particularly like this, even if they would use statcheck in the future.

The only real problem here was statcheck's authors publishing all the results and making a great deal of noise about it. They didn't have to do that, it is an aggressive move that was not designed to help authors, it was designed for statcheck to get attention. We have no idea how big of a problem it is, this article might have been mostly muckraking, and statcheck might be great and well liked.

Anyway, I don't think the system is broken. It is currently working better than it has worked at any time in the past, and it is continuing to improve. Statcheck might improve it more, but that remains to be seen. Other software tools already have improved it.

Re: A bot crawled thousands of studies looking for simple math errors

#84
post #82
post #66

Earlier quoted context omitted.

There is a third option that's better than either one you proposed. Get people writing new papers to use statcheck before they publish. The only rudeness here was an avoidable choice - publishing statcheck's results on a huge set of already published papers. The statcheck authors chose to do that for exposure, they even said so, it was not for posterity or the scientific well being of the community. I don't personall…

And what happens when a new fact-checking algorithm comes out? What I said in my comment.

Only if they're also rude about it and run it on a huge set of old papers, and do a big PR campaign to get attention.

Otherwise, the only thing that happens is papers quietly get better and everyone on all sides is happy.

Re: A bot crawled thousands of studies looking for simple math errors

#85
post #43
post #34

> and whether online critiques of past work constitute “bullying” or shaming. The PubPeer comments are just a tiny part of that debate. But it can hit a nerve. > Susan Fiske, a former president of the Association for Psychological Science, alluded to Statcheck in an interview with Business Insider, calling it a “gotcha algorithm.” > The draft of the article said these online critics were engaging in “methodological t…

I took a class with Susan Fiske in grad school, and the following is a quote from one of her comments on a homework where I had called into question a paper that was clearly not just statistically incorrectly done but was based on urban-legend-level assumptions: "These results are from respected investigators, so it's inappropriate to question them."

[deleted]

Re: A bot crawled thousands of studies looking for simple math errors

#86
post #77
post #34

> and whether online critiques of past work constitute “bullying” or shaming. The PubPeer comments are just a tiny part of that debate. But it can hit a nerve. > Susan Fiske, a former president of the Association for Psychological Science, alluded to Statcheck in an interview with Business Insider, calling it a “gotcha algorithm.” > The draft of the article said these online critics were engaging in “methodological t…

> If these are attitudes typical of psychology, then I cannot say I consider psychology to be a proper social science. It's quite unscientific to draw a blanket conclusion like that about a field based on the anecdotal evidence of one named person and some unnamed "critics". Let's get some statistics before casting doubt on the entire field of psychology. I think most people are trying to get by and to be good. My pe…

> If these are attitudes typical of psychology, then I cannot say I consider psychology to be a proper social science.

is a proposition. He/she's not concluding psychology is not a proper social science; he/she's stating

> If these are attitudes typical of psychology, then I cannot say I consider psychology to be a proper social science.

Re: A bot crawled thousands of studies looking for simple math errors

#87
post #77
post #34

> and whether online critiques of past work constitute “bullying” or shaming. The PubPeer comments are just a tiny part of that debate. But it can hit a nerve. > Susan Fiske, a former president of the Association for Psychological Science, alluded to Statcheck in an interview with Business Insider, calling it a “gotcha algorithm.” > The draft of the article said these online critics were engaging in “methodological t…

> If these are attitudes typical of psychology, then I cannot say I consider psychology to be a proper social science. It's quite unscientific to draw a blanket conclusion like that about a field based on the anecdotal evidence of one named person and some unnamed "critics". Let's get some statistics before casting doubt on the entire field of psychology. I think most people are trying to get by and to be good. My pe…

Sure "but not ALL psychologists!" is valid here (as it is in literally any discussion about anything online or elsewhere) but she is not just one person. She is the former head of a professional organization for psychologists; if we look for someone to represent them as a whole, she is the first place we should look. Would the professional organization for physics have as their head someone who believes we had it right when we said the earth was flat and the expertise of these scientists shouldn't be questioned?

How many psychologists have/will publicly denounced her? HN is ready to attack every police officer who hides and defends the actions of a few bad apples as bad apples themselves (rightly so), where's the equal treatment for psuedoscientists?

Re: A bot crawled thousands of studies looking for simple math errors

#88

Earlier quoted context omitted.

A fact is a human construct. It is a symbol. You just conflated facts with reality. A cursory examination of reality will reveal that observations of reality are observer dependent. So what is an objective fact? An objective fact must be one that is covariant in change of observer interpreting the fact. Physical laws like General Relativity are covariant in this way: the facts of relativity differ symbolically depend…

Ok. So this is semantics and in this science there is a different meaning for the word "fact." What is the utility of redefining a fact as a human construct of accepted information, rather than as data points corresponding to components of objective reality?

I think it's important to think about this because we as humans have major limitations in our understanding. For example, we are limited and biased due to the spatial and temporal scales of our senses, and we may be similarly affected by social things such as language, personal beliefs, interpersonal interactions.

Re: A bot crawled thousands of studies looking for simple math errors

#89
post #27

Earlier quoted context omitted.

This appears to be about STEM education, not STEM fields directly.

So STEM curriculum should teach there are no objective truths? 1 + 1 is not always 2? Failure to reproduce a study could just be the result of different cultural, gender, or racial differences in the researchers and does not necessarily invalidate anything?

I won't go so far as to agree there are no objective truths but your second point certainly sounds plausible. Researcher bias is a well known issue.

Re: A bot crawled thousands of studies looking for simple math errors

#90
post #34

> and whether online critiques of past work constitute “bullying” or shaming. The PubPeer comments are just a tiny part of that debate. But it can hit a nerve. > Susan Fiske, a former president of the Association for Psychological Science, alluded to Statcheck in an interview with Business Insider, calling it a “gotcha algorithm.” > The draft of the article said these online critics were engaging in “methodological t…

> That anyone in the field of psychology would even be comfortable publically espousing a non-scientific worldview like that means that psychologists are not being properly educated in the scientific method. This is a generalization about psychologists from a sample size of one psychologist, to suggest that they don't understand science?

One ex-president of the psychologists professional association, a person who was literally deliberately selected by the community of psychologists to represent them.

It would similarly be completely inappropriate to generalize the policy positions of Hillary Clinton to the Democratic party, yes?

Post reply on HN