A bot crawled thousands of studies looking for simple math errors • Hacker News Board

Re: A bot crawled thousands of studies looking for simple math errors

#121

Post by pbhjpbhj » Wed, Oct 05, 2016, 9:58 PM UTC

nitrogen wrote: post #62

nicops wrote: post #56

Earlier quoted context omitted.

> If you need to a 'chill' with the original researcher to get critical details, that's not science anymore. Certainly there is always the possibility that there was some details misunderstood, something that needs to be clarified, a print error, etc. Your "that's not science anymore" statement seems highly exaggerated. People are not supposed to communicate only via papers.

The usefulness of a paper that doesn't stand on its own is rather limited, though.

All papers should start with a dictionary? No clearly not, so there's always going to be some assumed knowledge - words change their meaning and have different meaning to different people so we're already on to a loser just with the medium we're using.

So, the possibility of things like, say, a researcher not mentioning something that is standard practice in their lab that later is found to be a crucial part of the setup for an experiment seems high. But just like you don't want to provide a dictionary of standard terms with a paper you don't want to provide a list of the chemicals used to mop the floor, or a list of the lumen and colour temperature ratings of lights in the fume cupboards, or ...

IMO if a paper is not reproducible then yes it should be published but also the original team producing the paper should be challenged to reproduce the results. It's not a fight, we're all on the same team - work with them and try to find the reason for the lack of reproducibility.

Re: A bot crawled thousands of studies looking for simple math errors

#122

Post by dahart » Wed, Oct 05, 2016, 9:58 PM UTC

int_19h wrote: post #107

dahart wrote: post #84

Earlier quoted context omitted.

Only if they're also rude about it and run it on a huge set of old papers, and do a big PR campaign to get attention. Otherwise, the only thing that happens is papers quietly get better and everyone on all sides is happy.

So they shouldn't run it on the old papers, so as to not offend anyone's sensibilities? Do we just declare said papers completely useless then? Or do they keep getting cited by new papers, and used to guide policy? If the latter, then not vetting them using the newer and better methodology would be unethical.

> If the latter, then not vetting them using the newer and better methodology would be unethical.

On the contrary, it would be unethical to hold people to new standards that didn't exist when the work was done. If you have a beer and then next week the laws change the drinking age to 65 years old, should you go to jail?

This is simply not how things are done, in science or in society. Laws and standards change all the time, you are only subject to the laws or standards that exist when the work or action you do is performed & evaluated. For the purposes of scientific publication, we do not and will never revisit all prior work and formally re-judge whenever standards or policies change.

You may be conflating publication policy with general scientific understanding. Old papers will always be informally evaluated under the thinking of the day. But that doesn't help the old papers, nothing can be done about the old papers, they are part of a fixed record that can't change, it only allows us to publish new papers. What will and does happen now is new papers will be published that refute old papers. The new papers are subject to the new methodologies.

Re: A bot crawled thousands of studies looking for simple math errors

#123

Post by jessaustin » Wed, Oct 05, 2016, 9:59 PM UTC

dmd wrote: post #43

c3534l wrote: post #34

> and whether online critiques of past work constitute “bullying” or shaming. The PubPeer comments are just a tiny part of that debate. But it can hit a nerve. > Susan Fiske, a former president of the Association for Psychological Science, alluded to Statcheck in an interview with Business Insider, calling it a “gotcha algorithm.” > The draft of the article said these online critics were engaging in “methodological t…

I took a class with Susan Fiske in grad school, and the following is a quote from one of her comments on a homework where I had called into question a paper that was clearly not just statistically incorrectly done but was based on urban-legend-level assumptions: "These results are from respected investigators, so it's inappropriate to question them."

Haha this reminds me of the feedback I got on a report in Comparative Religion, when I questioned some French dude's really elaborate and fanciful interpretation of the paintings at Lascaux. "Ummm... what supporting texts do we have from 17,000 years ago?" "No, you can't ask that sort of question and it's really disrespectful."

FWIW, I wasn't really upset by that feedback. Senior spring; I don't care!

Re: A bot crawled thousands of studies looking for simple math errors

#124

Post by tormeh » Wed, Oct 05, 2016, 10:00 PM UTC

1_listerine_pls wrote: post #41

cafebeen wrote: post #28

Earlier quoted context omitted.

Well, scientists are people too, so they have feelings and are not perfect. I think there are polite ways provide criticism and corrections, hopefully without humiliating people with good intentions. If it's a simple mistake, they can email the author and suggest an erratum. If it's more serious, then it's common to write a response in the journal. I think more public forms of criticism are perceived as bullying beca…

I don't care about their reputations. It's not a school mascot contest. These are scientific papers.

Yes, and scientists don't care. Science, as it has been explained to me by scientists, is a penis length contest, where impact factor and prestige serve as ersatz penises.

Re: A bot crawled thousands of studies looking for simple math errors

#125

Post by alphonsegaston » Wed, Oct 05, 2016, 10:03 PM UTC

zeveb wrote: post #76

alphonsegaston wrote: post #60

Earlier quoted context omitted.

Aside from things like pronouns, English hasn't been gendered since the 14th century: https://en.m.wikipedia.org/wiki/Gender_in_English

'Aside from that Mrs. Lincoln, how was the play?' Pronouns are precisely what folks who complain about gender in English are complaining about (well, that and words like 'mankind').

Glibness aside, the usage of gendered pronouns does not make a language gendered. That's only true if there has to be agreement between the gender of a noun and words relating to the noun.

And it's also not what people are arguing. Instead, they argue that gendered terms are the product of cultural norms and prescriptive grammar that reinforce gender roles that are oppressive to both men and women. That some view these assignments as objective when that's untrue linguistically and historically, I think only lends credence to their point.

Re: A bot crawled thousands of studies looking for simple math errors

#126

Post by Practicality » Wed, Oct 05, 2016, 10:09 PM UTC

mentat wrote: post #105

merpnderp wrote: post #7

Earlier quoted context omitted.

Objectivity is listed as a problem with science curriculum. Welcome to the future. http://nsuworks.nova.edu/cgi/viewcontent.cgi?article=2467&co...

Based on my experience with feminist though in academia, objectivity as a mask for bias is the issue. The preferred method to address is to surface as much of the implicit bias a possible so the whole can be considered. I don't find this problematic.

Bias is a big problem, but why not call out the problem directly instead of confusing the issue with a tangential, philosophical discussion about whether or not objectivity exists?

Re: A bot crawled thousands of studies looking for simple math errors

#127

Post by law_of_the_hobo » Wed, Oct 05, 2016, 10:20 PM UTC

dmd wrote: post #43

c3534l wrote: post #34

> and whether online critiques of past work constitute “bullying” or shaming. The PubPeer comments are just a tiny part of that debate. But it can hit a nerve. > Susan Fiske, a former president of the Association for Psychological Science, alluded to Statcheck in an interview with Business Insider, calling it a “gotcha algorithm.” > The draft of the article said these online critics were engaging in “methodological t…

I took a class with Susan Fiske in grad school, and the following is a quote from one of her comments on a homework where I had called into question a paper that was clearly not just statistically incorrectly done but was based on urban-legend-level assumptions: "These results are from respected investigators, so it's inappropriate to question them."

Out of curiosity, do you remember what paper it was?

Re: A bot crawled thousands of studies looking for simple math errors

#128

Post by int_19h » Wed, Oct 05, 2016, 10:26 PM UTC

dahart wrote: post #122

int_19h wrote: post #107

Earlier quoted context omitted.

So they shouldn't run it on the old papers, so as to not offend anyone's sensibilities? Do we just declare said papers completely useless then? Or do they keep getting cited by new papers, and used to guide policy? If the latter, then not vetting them using the newer and better methodology would be unethical.

> If the latter, then not vetting them using the newer and better methodology would be unethical. On the contrary, it would be unethical to hold people to new standards that didn't exist when the work was done. If you have a beer and then next week the laws change the drinking age to 65 years old, should you go to jail? This is simply not how things are done, in science or in society. Laws and standards change all th…

You're not holding people to new standards. You're holding their research to new standards. Which is the only sensible course of action.

And no-one suggested that those papers should be rewritten. But insofar as there's some known problem with them, why is it a bad thing to have a public record stating that much?

Re: A bot crawled thousands of studies looking for simple math errors

#129

Post by [deleted] » Wed, Oct 05, 2016, 10:35 PM UTC

tormeh wrote: post #124

1_listerine_pls wrote: post #41

Earlier quoted context omitted.

I don't care about their reputations. It's not a school mascot contest. These are scientific papers.

Yes, and scientists don't care. Science, as it has been explained to me by scientists, is a penis length contest, where impact factor and prestige serve as ersatz penises.

[deleted]

Re: A bot crawled thousands of studies looking for simple math errors

#130

Post by hueving » Wed, Oct 05, 2016, 10:45 PM UTC

6stringmerc wrote: post #19

What a clever and, dare I say it, fantastically useful experiment! So much less harm than even "door knob twisting" type explorations - no, this was using published works and pretty much running them through a process to verify or not verify accuracy. Unsolicited? So what! As a practiced writer I make unsolicited judgments on language usage all the time. Are these people that completely write from their own minds and…

Putting grammar on the same level as this is wrong. Poor grammar and spelling doesn't frequently change the entire results of a paper. Math errors though can easily be the difference between significance and insignificance, changing the entire set of conclusions that can be drawn from the paper.