Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

A bot crawled thousands of studies looking for simple math errors

vox.com

Post reply on HN
Print view View original

181–190 of 221 posts

Re: A bot crawled thousands of studies looking for simple math errors

#181

Post by [deleted] »

yakult wrote: post #162
mentat wrote: post #101

Earlier quoted context omitted.

> People are not supposed to communicate only via papers. That we use written communication that persists through generations is the basis of science and society in general. If we cannot communicate sufficiently via papers, we're in a world of trouble.

I have a concrete counterexample. Let's say I write a paper presenting a model, plus some numerical results of large simulations. The code is based on gluing together various pieces of open source code. All these codes are typical scientist codes that are held together with duct tape. My paper is short, but I spent a lot of effort munging things together, and I'm fairly certain nobody can reproduce my results without…

[deleted]

Re: A bot crawled thousands of studies looking for simple math errors

#182

Post by adrianN »

yakult wrote: post #168
erichocean wrote: post #167

Earlier quoted context omitted.

Open source the code and supporting data.

It's not that simple. What if some of it is proprietary? What if I'm not allowed to submit code because I need to be anonymous so reviewers can maintain impartiality? What happens when one of the upstreams update and breaks my code? Do I need to keep it updated? Forever?

At my institute at least, scientists are required to maintain everything that is necessary to reproduce a result for at least ten years. That includes all the data and the software used to produce the results. It's not an easy job, but it's important.

Re: A bot crawled thousands of studies looking for simple math errors

#183

Post by danso »

M_Grey wrote: post #176
c3534l wrote: post #34

> and whether online critiques of past work constitute “bullying” or shaming. The PubPeer comments are just a tiny part of that debate. But it can hit a nerve. > Susan Fiske, a former president of the Association for Psychological Science, alluded to Statcheck in an interview with Business Insider, calling it a “gotcha algorithm.” > The draft of the article said these online critics were engaging in “methodological t…

If psychologists were educated properly to be scientists, they'd abandon psychology, which is at best an art.

I agree with Dr. Srivastava that psychology is not only a science but the hardest science:

https://hardsci.wordpress.com/2009/03/14/making-progress-in-...

> What are the “hard” — as in difficult — problems in science? Hard problems in science are those that are embedded in complex systems; they are hard because to study something well you often need to isolate it from outside influences. Hard problems are those that vary by local conditions — science seeks to identify general laws, and when something is locally dependent, you need to sniff out the complex interactions that make it so. And hard problems are those that are difficult to quantify — science rests upon formalization and quantification, and you need to get traction at that initial step of quantification (i.e., measurement) before you can test theories. So… by these measures, if we are going to differentiate areas of science, the continuum of scientific problems should go from “hard” to “easy,” and psychology is clearly a science that deals with hard problems. Perhaps the hardest.

Reminds me of the counterintuitive about the two hard problems in computer science. Naming things, and deciding when data is obsolete, are easy mechanically. But because they have neither a deterministic nor correct solution, they end up being the hard problems you face as you build bigger projects. Or, to emphasize a line from Dr. Srivastava above:

> And hard problems are those that are difficult to quantify — science rests upon formalization and quantification, and you need to get traction at that initial step of quantification (i.e., measurement) before you can test theories.

It's possible for a science to be "hard" while having a lot of mediocre practitioners.

Re: A bot crawled thousands of studies looking for simple math errors

#184

Post by yakult »

adrianN wrote: post #182
yakult wrote: post #168

Earlier quoted context omitted.

It's not that simple. What if some of it is proprietary? What if I'm not allowed to submit code because I need to be anonymous so reviewers can maintain impartiality? What happens when one of the upstreams update and breaks my code? Do I need to keep it updated? Forever?

At my institute at least, scientists are required to maintain everything that is necessary to reproduce a result for at least ten years. That includes all the data and the software used to produce the results. It's not an easy job, but it's important.

If your institute also mandates they make the data/software publicly available, then that's definitely the exception rather than rule. Also must be hideously expensive.

It almost never happens that a paper I read actually comes with usable source code.

Re: A bot crawled thousands of studies looking for simple math errors

#185

Post by Consultant32452 »

oldmanjay wrote: post #51
Consultant32452 wrote: post #46

Earlier quoted context omitted.

Yeah, this is an abusive marketing technique. If you don't want future embarrassment you'd better buy our service.

Or just be more vigilant and less defensive, both of which improve the practice of science

That's just not how polite human interaction works. If you become aware that someone has made a mistake you inform them privately to give them an opportunity to rectify that mistake. In the case of a scientific journal entry you might instead publish a response article. You don't put everyone on blast and practically blackmail them into purchasing your for-profit service. It was never about the science at all, it was about the profits for this product/service.

Re: A bot crawled thousands of studies looking for simple math errors

#186

Post by mcbits »

hueving wrote: post #160
6stringmerc wrote: post #146

Earlier quoted context omitted.

Uh, no, poor grammar or words or phrases used improperly can genuinely have cascade / domino effects. Brush up on "Eats, Shoots, and Leaves" if you wish to counter-balance your math-centric logic notions with how similar principles apply in written language. English is, without question, one of the more tricky (re: confusing) mutts of linguistics.

Can you point me to a published paper where the entire meaning changed because of poor grammar?

I don't have an example from a published academic paper, but from the linked article:

> the odds are this will stoke real hostility for those who are already dubious about what has been termed 'bullying' and so on by people interested in reproducibility.

I'm pretty sure this was intended to be read as

    [has been termed] ['bullying' and so on by people interested in reproducibility].
But it could also be read as

    [has been termed 'bullying' and so on] [by people interested in reproducibility].
Ambiguous wording like this could easily be overlooked in review, especially if the misinterpretation agrees with what the reader was already expecting.

Re: A bot crawled thousands of studies looking for simple math errors

#187

Post by Klockan »

yakult wrote: post #162
mentat wrote: post #101

Earlier quoted context omitted.

> People are not supposed to communicate only via papers. That we use written communication that persists through generations is the basis of science and society in general. If we cannot communicate sufficiently via papers, we're in a world of trouble.

I have a concrete counterexample. Let's say I write a paper presenting a model, plus some numerical results of large simulations. The code is based on gluing together various pieces of open source code. All these codes are typical scientist codes that are held together with duct tape. My paper is short, but I spent a lot of effort munging things together, and I'm fairly certain nobody can reproduce my results without…

> Let's say I write a paper presenting a model, plus some numerical results of large simulations. The code is based on gluing together various pieces of open source code. All these codes are typical scientist codes that are held together with duct tape. My paper is short, but I spent a lot of effort munging things together, and I'm fairly certain nobody can reproduce my results without my source code (preferably the whole environment) unless they spend a lot of time on trial and error like I did.

Then leave out the results since they are just an anecdote. If you want to include experimental results then it has to be done in a scientific fashion.

Re: A bot crawled thousands of studies looking for simple math errors

#188

Post by V-2 »

jessaustin wrote: post #123
dmd wrote: post #43

Earlier quoted context omitted.

I took a class with Susan Fiske in grad school, and the following is a quote from one of her comments on a homework where I had called into question a paper that was clearly not just statistically incorrectly done but was based on urban-legend-level assumptions: "These results are from respected investigators, so it's inappropriate to question them."

Haha this reminds me of the feedback I got on a report in Comparative Religion, when I questioned some French dude's really elaborate and fanciful interpretation of the paintings at Lascaux. "Ummm... what supporting texts do we have from 17,000 years ago?" "No, you can't ask that sort of question and it's really disrespectful." FWIW, I wasn't really upset by that feedback. Senior spring; I don't care!

It immediately reminded me of "Emperor Kennedy Legend: A New Anthropological Debate" by Leszek Kołakowski : ) Highly recommended and very funny snce there's much truth to it I think. I found it online in English: http://jennis22.weebly.com/uploads/3/0/5/3/30531242/emperor_...

Re: A bot crawled thousands of studies looking for simple math errors

#190

Post by Bartweiss »

c3534l wrote: post #34

> and whether online critiques of past work constitute “bullying” or shaming. The PubPeer comments are just a tiny part of that debate. But it can hit a nerve. > Susan Fiske, a former president of the Association for Psychological Science, alluded to Statcheck in an interview with Business Insider, calling it a “gotcha algorithm.” > The draft of the article said these online critics were engaging in “methodological t…

Calling Statcheck a "gotcha algorithm" descends into self-parody. It reads like outrage at the idea that anyone might expect you to be accurate about facts.
Post reply on HN