Live data from Hacker News

Twenty-nine teams use same dataset, find contradicting results [pdf]

osf.io

21–30 of 36 posts

Re: Twenty-nine teams use same dataset, find contradicting results [pdf]

#21
post #11

Reminds me of the idea (Robin Hanson's, I think?) to add an extra layer of blindness to studies: during peer review, take the original data, and write a separate paper with the opposite conclusion. Randomize which reviewers get which version. Your original paper is then only accepted if they reject the inverted version.

I think you misremembered it: http://www.overcomingbias.com/2007/01/conclusionblind.html http://www.overcomingbias.com/2010/11/results-blind-peer-rev... Nothing about accepted only if they rejected the reversed version; just that the pro & con versions be supplied (first post), or a paper sans conclusions/results (second post).

Re: Twenty-nine teams use same dataset, find contradicting results [pdf]

#22
post #19
post #8

Earlier quoted context omitted.

Off-topic, but you seem well positioned to answer: Why do you say "TL:DR" here when summarizing a short blog post that you enjoyed? Clearly the meaning has diverged from the original abbreviated insult of "Too long; didn't read", but I don't understand what people mean when they use it today. Why did you phrase it this way? Are you a native English speaker? If not intended to be derogatory, does the dissonance bother…

How do you see it as derogatory? I'm a native English speaker and have never thought of it that way. I didn't click the link, but did appreciate his short summary – and upvoted him for it. ;)

'tl;dr' is often a troll response to a long post that that someone has obviously spent a lot of time on. Bonus troll-points if the long post was in response to another troll.

Example:

poster1: only retards use vi, notepad rules

poster2: huge list of reasons why vi is better than notepad

poster1: lol tldr

Re: Twenty-nine teams use same dataset, find contradicting results [pdf]

#23
post #14

I understand how tempting it is in our age of big data and all that stuff to perceive this as some curious new phenomena, but it really is not. This is precisely the reason why we've come up with some criteria for "science" quite a while ago. And in fact, all this experiment is pretty meaningless. So, for starters: 29 students get the same question on the math/physics/chemistry exam and give 29 different answers. Bre…

In physics they can do experiments to get highly confident results. In medicine, economics, and any other science dealing with people, data is much harder to collect and there are harder ethics involved. We could learn so much if we simply performed controlled experiments on the global economy, politics be damned! Or if we could just make the professionals play more soccer matches in a controlled setting (but just like a pro match in every other regard!). Or if we were more aggressive with human trials of drugs. But we can't. Scientists in some fields are stuck with data sets where they'll never get 5 sigma confidence. Does that mean they should stop using statistics? Hell no. There are still very useful things to be learned. It's just much harder to get right.

Re: Twenty-nine teams use same dataset, find contradicting results [pdf]

#24
post #19
post #8

Earlier quoted context omitted.

Off-topic, but you seem well positioned to answer: Why do you say "TL:DR" here when summarizing a short blog post that you enjoyed? Clearly the meaning has diverged from the original abbreviated insult of "Too long; didn't read", but I don't understand what people mean when they use it today. Why did you phrase it this way? Are you a native English speaker? If not intended to be derogatory, does the dissonance bother…

How do you see it as derogatory? I'm a native English speaker and have never thought of it that way. I didn't click the link, but did appreciate his short summary – and upvoted him for it. ;)

I also appreciated the summary; my question was just about the phrasing. In it's literal usage, it's saying that the article had nothing useful to say: http://knowyourmeme.com/memes/tldr.

That clearly wasn't the case here, so I was wondering why the author choose to use it. I realize that meaning has changed over time, but I was wondering what meaning he (and others) intend when it is used.

Re: Twenty-nine teams use same dataset, find contradicting results [pdf]

#26
post #14

I understand how tempting it is in our age of big data and all that stuff to perceive this as some curious new phenomena, but it really is not. This is precisely the reason why we've come up with some criteria for "science" quite a while ago. And in fact, all this experiment is pretty meaningless. So, for starters: 29 students get the same question on the math/physics/chemistry exam and give 29 different answers. Bre…

I don't understand what it is that you want to say, even if we were to accept all of your premises. That other than math and physics we actually know nothing? That knowledge (even if partial) is meaningless unless it is as rigorous as the one we can attain in physics, the simplest of sciences? Obviously we can be less confident of results in the complex/intractable/inexact sciences than we can in the simple/tractable/exact ones. You want to call only the latter group "science" and the former something else? Fine. Does that mean we should completely ignore all results in disciplines which aren't science?

Re: Twenty-nine teams use same dataset, find contradicting results [pdf]

#27
post #8
post #3

Earlier quoted context omitted.

The blog post is a great overview as well as useful context, thanks for sharing it. TL:DR summary: Scientific results are highly contingent on subjective decisions at the analysis stage. Different (well-founded) data analysis techniques on a fairly simple and well-defined problem can give radically different results. It's very interesting research -- a great real-life example supporting the models Scott Page et. al.…

Off-topic, but you seem well positioned to answer: Why do you say "TL:DR" here when summarizing a short blog post that you enjoyed? Clearly the meaning has diverged from the original abbreviated insult of "Too long; didn't read", but I don't understand what people mean when they use it today. Why did you phrase it this way? Are you a native English speaker? If not intended to be derogatory, does the dissonance bother…

Like 'justinlardinois I just use "TL:DR summary" to mean a "short summary." I didn't mean it as derogatory, and it doesn't feel dissonant to me -- although given how you view it I can see why you do. And yes, I'm a native English speaker.

Re: Twenty-nine teams use same dataset, find contradicting results [pdf]

#28
post #4

Lies, damned lies, and statistics https://en.wikipedia.org/wiki/Lies,_damned_lies,_and_statist... Statistics can be manipulated surprisingly easily.

There are three kinds of lies. There are also three kinds of comments I see in this thread: > "This is interesting, here's some thoughts and ideas that further contribute to this subject" > "This is interesting, here's a link to some further writing on this subject" > "The entire concept and discipline of statistics is bullshit." Par for the course here at Hacker News.

4. > "About TL:DR ...:"

Also par for the course here :)

Re: Twenty-nine teams use same dataset, find contradicting results [pdf]

#29
post #22
post #19

Earlier quoted context omitted.

How do you see it as derogatory? I'm a native English speaker and have never thought of it that way. I didn't click the link, but did appreciate his short summary – and upvoted him for it. ;)

'tl;dr' is often a troll response to a long post that that someone has obviously spent a lot of time on. Bonus troll-points if the long post was in response to another troll. Example: poster1: only retards use vi, notepad rules poster2: huge list of reasons why vi is better than notepad poster1: lol tldr

> "'tl;dr' is often..."

But not always. You have to consider how it was used to tell whether it was meant with ill will, the term tl;dr on its own doesn't necessarily tell you enough.

Re: Twenty-nine teams use same dataset, find contradicting results [pdf]

#30

So, does this mean every team used improper methodology? Or can we meta-review the results and figure out what's really going on?

It makes it really hard to work out what's happening, especially if you want the result to match existing standards.

For a real world example of this see deworming schoolchildren.

People looking at the educational effects of deworming children reach different conclusions because some of them use a medical model and some of them use an economics model.

http://www.cochrane.org/news/educational-benefits-deworming-...

http://www.cochrane.org/CD000371/INFECTN_deworming-school-ch...

Talked about in this More or Less episode:

http://www.bbc.co.uk/programmes/b0659q1f

http://www.theguardian.com/society/2015/jul/23/research-glob...

Post reply on HN