Live data from Hacker News

A bot crawled thousands of studies looking for simple math errors

vox.com

171–180 of 221 posts

Re: A bot crawled thousands of studies looking for simple math errors

#171
post #34

> and whether online critiques of past work constitute “bullying” or shaming. The PubPeer comments are just a tiny part of that debate. But it can hit a nerve. > Susan Fiske, a former president of the Association for Psychological Science, alluded to Statcheck in an interview with Business Insider, calling it a “gotcha algorithm.” > The draft of the article said these online critics were engaging in “methodological t…

"a proper social science" Is 'social science' even a 'proper science' ?

In general, if you have to put science in the name, it isn't.

Re: A bot crawled thousands of studies looking for simple math errors

#172
post #124

Earlier quoted context omitted.

I don't care about their reputations. It's not a school mascot contest. These are scientific papers.

Yes, and scientists don't care. Science, as it has been explained to me by scientists, is a penis length contest, where impact factor and prestige serve as ersatz penises.

The best part is that any fool can see that impact factors are a terrible proxy for real impact, and in fact the editors of several glam journals have pointed this out (disclosure: I have plenty of glam papers on my CV*).

But since it props up a myth that senior faculty like to believe (i.e. their shitty old Cell paper is great because Cell is/was great) that's the yardstick. As the director of a CCC once told me, it takes too long to see the impact of papers (citations piling up) so the JIF is used as a proxy.

Sort of like how it takes too long to do good science, so some people just publish whatever garbage they can sneak past the editors (ha fucking ha only serious). There is a LOT of crap in the literature as a result.

Re: A bot crawled thousands of studies looking for simple math errors

#173

Earlier quoted context omitted.

I find it hard to understand how something that is clearly an automated bot, posting comments that are 100% factual ("I checked this paper; N things looked wrong"), could be perceived as humiliating. I guess it would be the case if you assumed that all people are perfect and never make mistakes. I would hope that psychologists, of all people, would know better than that. So if it's not that, where's the humiliation p…

I don't necessarily think this is bullying or humiliating, but it's silly to think that being done by a "bot" and being "factual" has anything to do with it. If a malware "bot" secretly posted people's porn viewing history to their facebook page, would that not be humiliating for those people? Or would it not because it was factual and done by a "bot"? Clearly it would be. Saying it was done by a bot is no excuse for…

Being done by a bot is not an excuse in general. However, when pointing out objective factual mistakes, I think there is a difference between your colleague pointing it out, and an automated tool pointing it out. Even if result is the same, the former can be embarrassing, hence why we learn to phrase negative responses in a roundabout way. But politeness cannot be expected of a brainless machine, and so it can deliver simple facts.

So it seems that it boils down to public disclosure before private?

Out of curiosity, how would you imagine the "correct the mistakes" procedure after private disclosure? The author cannot just edit the paper, it's already published. They would have to publish errata, which draws just as much attention. And, from an ethical perspective, if an author is notified of a mistake found by autonomous tool, wouldn't they be required to disclose the methodology when publishing errata? So I'm not sure how that whole situation is fundamentally different from just dumping it in the public.

Re: A bot crawled thousands of studies looking for simple math errors

#174
post #34

> and whether online critiques of past work constitute “bullying” or shaming. The PubPeer comments are just a tiny part of that debate. But it can hit a nerve. > Susan Fiske, a former president of the Association for Psychological Science, alluded to Statcheck in an interview with Business Insider, calling it a “gotcha algorithm.” > The draft of the article said these online critics were engaging in “methodological t…

As someone in this field somewhat close to the article in a certain sense, I wouldn't say Susan Fiske's attitudes are typical of psychology. Maybe not rare, but not typical either. Her comments have been very controversial to say the least, and many are disturbed by them. Moreover, if you level that statement against psychology, I'm afraid you have to level it against medicine as well, given that there have been similar sentiments expressed in major medical journals as well recently.

Also, without meaning to defend Fiske, her comments here are taken out of context somewhat. Her reference to "methodological terrorism" (from what I have heard through the grapevine) is more, or at least in part, about the trend toward having scientific debates outside of the peer review process, in social media. So my guess is that she might say that part of what she objects to about Statcheck is that it crawls through the papers, labels an error, and then we end up discussing it on HN rather than through peer review. What if Statcheck made an error, which it does sometimes? I don't agree with it, but I think the position I'm describing (which I think is her point in part) isn't unreasonable either. That is, it's not the checking of stats, it's the chaos and disintegration of the peer review system, and "extrascientific" discourse that's happening in science today, if you define "extrascientific" as "outside of peer reviewed journals," where your critics are attacking you on twitter, forums, and facebook, more so than in professional published outlets in a sort of mob.

Again, I do not share her perspective at all (I'm in favor of a shift away from journals) but I do think here her original point was twisted a bit.

Re: A bot crawled thousands of studies looking for simple math errors

#175

Earlier quoted context omitted.

I used to hold this opinion, but my experience with academic research changed my mind. Much of the scientific knowledge we have is passed from generation to generation by mentoring. The amount of knowledge is so vast, and our means of searching the written literature for relevant facts so poor, that when I want to learn something or solve a specific problem there is no substitute for a discussion with an expert in th…

>I have come to believe that the main purpose of hiring scientists in academia is to keep knowledge alive and have it passed on to future generations. Then these scientists should be devoted to producing textbooks and courses which can then be taught to non-research students. Yes, all knowledge about the scale of what a single individual knows (and keeps on their shelves, hard drives, etc) is embodied as communities…

> Then these scientists should be devoted to producing textbooks and courses ...

Textbooks and courses exist for everything but the most cutting edge stuff (which are still in flux anyway), but they are a very inefficient way of transferring knowledge. I would say they are practically useless without expert guidance. At the most basic level, there are so many of them that an expert has to tell you which ones are both good and relevant to what you want to learn. I've once seen a student waste months of his life studying a book he thought was relevant, only to discover that book wasn't building towards the sort of knowledge he needed in that subject. The book was about the correct subject, but was focused on somewhat different aspects than the ones he was interested in. There was no way for him to know this in advance without guidance.

So we don't know how to organize existing books. Also, even the books that exist are usually pretty bad at conveying knowledge. Or perhaps humans are just pretty bad at learning things from books. Either way, no one knows how to write textbooks and courses that are much better than what we have today. I really don't know of a better way to preserve knowledge than the current one. Perhaps technology can improve the situation by making access to knowledge more interactive. But I suspect this would require a real breakthrough.

> We can't afford to live in a world where only 1.7% know how things work.

Why not?

Re: A bot crawled thousands of studies looking for simple math errors

#176
post #34

> and whether online critiques of past work constitute “bullying” or shaming. The PubPeer comments are just a tiny part of that debate. But it can hit a nerve. > Susan Fiske, a former president of the Association for Psychological Science, alluded to Statcheck in an interview with Business Insider, calling it a “gotcha algorithm.” > The draft of the article said these online critics were engaging in “methodological t…

If psychologists were educated properly to be scientists, they'd abandon psychology, which is at best an art.

Re: A bot crawled thousands of studies looking for simple math errors

#177

What a clever and, dare I say it, fantastically useful experiment! So much less harm than even "door knob twisting" type explorations - no, this was using published works and pretty much running them through a process to verify or not verify accuracy. Unsolicited? So what! As a practiced writer I make unsolicited judgments on language usage all the time. Are these people that completely write from their own minds and…

I think "bullying" is a reductive term for what's actually happening. I think the point trying to be conveyed is that there is a certain decline in the author's personal status when they get holes punched in their publications.

As it should be, when you publish a stupid error that can substantially alter your conclusions.

Re: A bot crawled thousands of studies looking for simple math errors

#178

Earlier quoted context omitted.

Because you're putting the cart before the horse. Science is building a picture of """objective reality""" from human subjectivity. Data points aren't reality, they have to be interpreted. No matter how you try to do it, at some point you have to accept that a human being is doing science and the way science is done is fundamentally an intersubjective phenomenon. Science isn't reality. Scientific knowledge isn't real…

There's this funny thing I've noticed with your posts where the more words you use, the less utility I get from understanding the post. It's almost like you use overintellectualization to bury that you aren't saying much at all.

Clarity and brevity are the enemies of obfuscation.

Re: A bot crawled thousands of studies looking for simple math errors

#179

Earlier quoted context omitted.

Putting grammar on the same level as this is wrong. Poor grammar and spelling doesn't frequently change the entire results of a paper. Math errors though can easily be the difference between significance and insignificance, changing the entire set of conclusions that can be drawn from the paper.

Uh, no, poor grammar or words or phrases used improperly can genuinely have cascade / domino effects. Brush up on "Eats, Shoots, and Leaves" if you wish to counter-balance your math-centric logic notions with how similar principles apply in written language. English is, without question, one of the more tricky (re: confusing) mutts of linguistics.

> English is, without question, one of the more tricky (re: confusing) mutts of linguistics.

I am aware of two theories in linguistics about the complexity of languages:

1. All human languages are equally complex, presumably for some fundamental reason.

2. Some languages are more complex than others. Specifically, languages with a long history of being spoken only within isolated communities are more complex, and languages that have had a wave of non-native adults learn to speak them get simplified as that happens.

English is one of the go-to examples for simplicity in languages that have been learned by many adults; the others I'm aware of are Swahili (the trade language for north/east Africa), Mandarin Chinese (the cross-regional language for China), and (in reverse) Latin (the cross-regional language for western Europe -- a lot of grammatical features of Latin get lost as the Romance languages develop from it; you might wonder why Greek still has noun cases when the romance languages lost them).

I'm not aware of any theory that would suggest English is more tricky than other langauges.

Post reply on HN