Twenty-nine teams use same dataset, find contradicting results [pdf]
1–10 of 36 posts
Re: Twenty-nine teams use same dataset, find contradicting results [pdf]
#2A blog post giving background is at http://www.nature.com/news/crowdsourced-research-many-hands-....
Re: Twenty-nine teams use same dataset, find contradicting results [pdf]
#3A blog post giving background is at http://www.nature.com/news/crowdsourced-research-many-hands-... .
The blog post is a great overview as well as useful context, thanks for sharing it.
TL:DR summary: Scientific results are highly contingent on subjective decisions at the analysis stage. Different (well-founded) data analysis techniques on a fairly simple and well-defined problem can give radically different results.
It's very interesting research -- a great real-life example supporting the models Scott Page et. al. use for the value of cognitive diversity. The thrust of the blog post is about where crowdsourcing analysis can be helpful (as well as reasonable caveats about where it might not apply), which is certainly an interesting question. Obvioulsy, there are a lot of other implications to this as well.
Re: Twenty-nine teams use same dataset, find contradicting results [pdf]
#4Lies, damned lies, and statistics
https://en.wikipedia.org/wiki/Lies,_damned_lies,_and_statist...
Statistics can be manipulated surprisingly easily.
Re: Twenty-nine teams use same dataset, find contradicting results [pdf]
#5"The primary research question tested in the crowdsourced project was whether soccer players with dark skin tone are more likely than light skin toned players to receive red cards from referees."
This seems like a topic where one indeed typically winds-up with a multitude of competing conclusions.
Among other factors for we have:
* Pre-existing beliefs on the part of researchers.
* Lack of sufficient data.
* Difficulty in defining hypothese (is there a skin tone cut-off or should one look for degrees of skin tone and degrees of prejudice, should one look all referees or some referees).
Given this, I'd say it's a mistake to expect just numeric data at the level of complex social interactions to be anything like clear or unambiguous. If studies on topics such as this have value, they have to involve careful arguments concerning data collection, data normalization/massaging, and only then data analysis and conclusions.
But a lot of the context comes from prevalence shoddy studies that expect you can throw data in a bucket and draw conclusions, further facilitated having those conclusions echoed by mainstream media or by the media of one's chosen ideology.
Re: Twenty-nine teams use same dataset, find contradicting results [pdf]
#6This paper is awesome because it transparently folds the analytical approach into the experiment being conducted.
There are two kinds of scientific study: those where you can run another (ideally orthogonally approaching to the same question) experiment along with rigorous controls, and those where you can't.
The first type is much less likely to have results vary based on analytical technique (effectively the second experiment is a new analytical technique). Of course it does happen sometimes and sometimes the studies are wrong, still more controls and more experiments are always more better.
However, studies were you're limited by ethical or practical constraints (i.e. most experiments involving humans) don't have that luxury and therefore are far more contingent on decisions made at the analysis stage. What's awesome with this paper is it kind of gets around this limitation by trying different analytical methods, effectively each being a new "experiment" and seeing if they all reach the same consensus.
Interestingly, very few features in the analysis were shared among a large fraction of the teams, (only 2 features were used by more than 50% of teams) which suggests that no matter the method, the result holds true. A similar approach to open data and distributed analysis would be a really great way to eliminate some of the recent trouble with reproducibility in the broader scientific literature.
Re: Twenty-nine teams use same dataset, find contradicting results [pdf]
#7FiveThirtyEight did a write up of this paper (part 2):
http://fivethirtyeight.com/features/science-isnt-broken/
On the bright side, if you look at the 95CI for the 29 studies, almost all of them overlap.
Re: Twenty-nine teams use same dataset, find contradicting results [pdf]
#8A blog post giving background is at http://www.nature.com/news/crowdsourced-research-many-hands-... .
The blog post is a great overview as well as useful context, thanks for sharing it. TL:DR summary: Scientific results are highly contingent on subjective decisions at the analysis stage. Different (well-founded) data analysis techniques on a fairly simple and well-defined problem can give radically different results. It's very interesting research -- a great real-life example supporting the models Scott Page et. al.…
Off-topic, but you seem well positioned to answer: Why do you say "TL:DR" here when summarizing a short blog post that you enjoyed? Clearly the meaning has diverged from the original abbreviated insult of "Too long; didn't read", but I don't understand what people mean when they use it today. Why did you phrase it this way? Are you a native English speaker? If not intended to be derogatory, does the dissonance bother you?
Re: Twenty-nine teams use same dataset, find contradicting results [pdf]
#9Lies, damned lies, and statistics https://en.wikipedia.org/wiki/Lies,_damned_lies,_and_statist... Statistics can be manipulated surprisingly easily.
There are three kinds of lies. There are also three kinds of comments I see in this thread:
> "This is interesting, here's some thoughts and ideas that further contribute to this subject"
> "This is interesting, here's a link to some further writing on this subject"
> "The entire concept and discipline of statistics is bullshit."
Par for the course here at Hacker News.
Re: Twenty-nine teams use same dataset, find contradicting results [pdf]
#10Earlier quoted context omitted.
The blog post is a great overview as well as useful context, thanks for sharing it. TL:DR summary: Scientific results are highly contingent on subjective decisions at the analysis stage. Different (well-founded) data analysis techniques on a fairly simple and well-defined problem can give radically different results. It's very interesting research -- a great real-life example supporting the models Scott Page et. al.…
Off-topic, but you seem well positioned to answer: Why do you say "TL:DR" here when summarizing a short blog post that you enjoyed? Clearly the meaning has diverged from the original abbreviated insult of "Too long; didn't read", but I don't understand what people mean when they use it today. Why did you phrase it this way? Are you a native English speaker? If not intended to be derogatory, does the dissonance bother…
tl;dr stated off as a way of saying "this is too long and therefore requires too much effort for me to read it." Then that gave rise to people accompanying long reads with a "tl;dr version," which is usually a one or two sentence summary. Now that the latter is common and understood people just write tl;dr and then follow it with the summary, with the understanding that those who are unwilling to read the full version will read that instead.