Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

A bot crawled thousands of studies looking for simple math errors

vox.com

Post reply on HN
Print view View original

131–140 of 221 posts

Re: A bot crawled thousands of studies looking for simple math errors

#131

Post by cushychicken »

6stringmerc wrote: post #19

What a clever and, dare I say it, fantastically useful experiment! So much less harm than even "door knob twisting" type explorations - no, this was using published works and pretty much running them through a process to verify or not verify accuracy. Unsolicited? So what! As a practiced writer I make unsolicited judgments on language usage all the time. Are these people that completely write from their own minds and…

I think "bullying" is a reductive term for what's actually happening. I think the point trying to be conveyed is that there is a certain decline in the author's personal status when they get holes punched in their publications.

Re: A bot crawled thousands of studies looking for simple math errors

#132

Post by dahart »

int_19h wrote: post #128
dahart wrote: post #122

Earlier quoted context omitted.

> If the latter, then not vetting them using the newer and better methodology would be unethical. On the contrary, it would be unethical to hold people to new standards that didn't exist when the work was done. If you have a beer and then next week the laws change the drinking age to 65 years old, should you go to jail? This is simply not how things are done, in science or in society. Laws and standards change all th…

You're not holding people to new standards. You're holding their research to new standards. Which is the only sensible course of action. And no-one suggested that those papers should be rewritten. But insofar as there's some known problem with them, why is it a bad thing to have a public record stating that much?

We may be agreeing and mis-communicating, or agreeing violently, as I've heard it called, so let's get specific. What are you suggesting should happen to a paper when it's shown to have errors?

Nothing at all is wrong with adding new information to the public record stating the issues, that's what I mentioned is already happening -- new papers reference and demonstrate the weaknesses of old papers. In my field, as I suspect most, it's a time honored tradition to do a small literature review in the introduction section of a paper that mainly dismisses all related previous work as not good enough to solve the problem you're about to blow everyone's mind's with.

In my mind, nothing is wrong with what the statcheck authors did either. My one and only point at the top was that it's not surprising it ruffled some feathers, and that it didn't have to ruffle any feathers. That only happened because the results were made public without solicitation. @wyager was trying to paint the situation as a dichotomy between rude or unscientific, that rude was the only option. Rude is not the only option.

If statcheck hadn't published the review of old papers and contacted all the old authors, then I'm pretty sure two things would have happened: 1- this wouldn't have ruffled any feathers, and 2- it wouldn't have gotten much attention, and we wouldn't be talking about it.

Re: A bot crawled thousands of studies looking for simple math errors

#133

Post by vinchuco »

A good distinction between "peer reviewed" vs "computer verified"

>“The literature is growing faster and faster, peer review is overstrained, and we need technology to help us out,”

This is a problem in every field, not just Psychology.

I want someone to tell me the distribution (or average ratio) of papers read to papers written.

Every thesis written is supposed to add some delta to the state of the art. But there is no method for doing a diff between past and previous versions of human knowledge. How to make science less redundant and more efficient?

I dream of aggregators for everything.

Re: A bot crawled thousands of studies looking for simple math errors

#135

Post by dmd »

law_of_the_hobo wrote: post #127
dmd wrote: post #43

Earlier quoted context omitted.

I took a class with Susan Fiske in grad school, and the following is a quote from one of her comments on a homework where I had called into question a paper that was clearly not just statistically incorrectly done but was based on urban-legend-level assumptions: "These results are from respected investigators, so it's inappropriate to question them."

Out of curiosity, do you remember what paper it was?

I'm not entirely certain but I think it was

The modern synthesis of contrasting styles of thought in “east” and “west” *Nisbett, R. E., Masuda, T. (2003). Culture and point of view. Proceedings of the National Academy of Sciences, 100, September 16, 2003, 11163-11170

Re: A bot crawled thousands of studies looking for simple math errors

#136

Post by dahart »

wfo wrote: post #92
dahart wrote: post #15

Earlier quoted context omitted.

When I read that sentence, I immediately thought it sounded like some people were being childish and defensive. OTOH, a bunch of people got unsolicited error reports for already published papers, and I can understand being initially irritated. This was, by the admission of the statcheck authors, a way to get attention quickly. My guess is it'll all settle down and people won't complain about bullying once everyone ju…

In Mathematics, if you get an unsolicited error report and the report is accurate, your first reaction is "oh my god" as you assess the damage and how much revision needs to be done to correct the paper. Then the answer is "thank you! I didn't realize so many people even read my work so closely! I'll add a correction and thank you for your contribution in an update of the paper".

Yes! And I think this is true of most paper authors in most fields. It has been rare in my experience to see anyone end up angry or upset about getting a correction in the mail. The majority of people who publish know that errors occur all the time, and are glad to hear about it, more so if it happens early enough to do something about it.

I'd bet the majority of recipients of statcheck's automated correction, whatever their initial reaction, appreciate and end up wanting to use this kind of a tool before publishing their next paper.

It is worth mentioning there's a large stylistic difference between receiving an unsolicited error report directly from a reader & having a nice conversation about it, and being notified that an unsolicited error report has been published and attached to your paper automatically, without review, for all to see.

Re: A bot crawled thousands of studies looking for simple math errors

#137

Post by Cogito »

dahart wrote: post #122
int_19h wrote: post #107

Earlier quoted context omitted.

So they shouldn't run it on the old papers, so as to not offend anyone's sensibilities? Do we just declare said papers completely useless then? Or do they keep getting cited by new papers, and used to guide policy? If the latter, then not vetting them using the newer and better methodology would be unethical.

> If the latter, then not vetting them using the newer and better methodology would be unethical. On the contrary, it would be unethical to hold people to new standards that didn't exist when the work was done. If you have a beer and then next week the laws change the drinking age to 65 years old, should you go to jail? This is simply not how things are done, in science or in society. Laws and standards change all th…

I don't get this argument.

There is no new standard here. This is just a tool that says, "is there anything that looks like a mathematical/statistical mistake".

The expectation that a paper's calculations are correct and error free is one that has always existed.

There is no "thinking of the day", there is just mathematical correctness or not. At most you could argue that we might learn that some thing we thought mathematically true is no longer so, and warrants us reviewing papers where it had previously been used, but that is not the case here.

It is not unreasonable to hold published research to the standard of correctness, and if a paper contains errors within its calculations these should be fixed - regardless of when or who published it.

Re: A bot crawled thousands of studies looking for simple math errors

#138

Post by lutusp »

c3534l wrote: post #34

> and whether online critiques of past work constitute “bullying” or shaming. The PubPeer comments are just a tiny part of that debate. But it can hit a nerve. > Susan Fiske, a former president of the Association for Psychological Science, alluded to Statcheck in an interview with Business Insider, calling it a “gotcha algorithm.” > The draft of the article said these online critics were engaging in “methodological t…

> If these are attitudes typical of psychology, then I cannot say I consider psychology to be a proper social science.

In my experience, such attitudes are typical. Here's a 2014 paper by a Harvard psychology professor who tries to argue that null replication results shouldn't be taken seriously because the failure to replicate ipso facto indicts the study's methods:

http://jasonmitchell.fas.harvard.edu/Papers/Mitchell_failed_...

Quote: "Because experiments can be undermined by a vast number of practical mistakes, the likeliest explanation for any failed replication will always be that the replicator bungled something along the way ... The field of social psychology can be improved, but not by the publication of negative findings."

Re: A bot crawled thousands of studies looking for simple math errors

#139

Post by kaspiCZ »

pbhjpbhj wrote: post #121
nitrogen wrote: post #62

Earlier quoted context omitted.

The usefulness of a paper that doesn't stand on its own is rather limited, though.

All papers should start with a dictionary? No clearly not, so there's always going to be some assumed knowledge - words change their meaning and have different meaning to different people so we're already on to a loser just with the medium we're using. So, the possibility of things like, say, a researcher not mentioning something that is standard practice in their lab that later is found to be a crucial part of the s…

> So, the possibility of things like, say, a researcher not mentioning something that is standard practice in their lab

I'd suggest a different formulation: "standard practice in their field"

Standard practice in general cooking? That's ok. Standard practice in my kitchen? That's a problem.

The research is IMO like a meal recipe a knowledgable chef should be able to reproduce.

Though it is understandable why one would forget to mention something. Especially if they thought it was general practice to do something their way.

Post reply on HN