Statistical challenges and misreadings of literature create unreplicable science [pdf]
1–10 of 57 posts
Re: Statistical challenges and misreadings of literature create unreplicable science [pdf]
#2On page 11 there is a mention of taking the result of self reporting (surveys) at their word. I’ve wondered about this issue not just in science but other situations. For example political polling, data point in time surveys, census, etc. Without verification, what good is the data? And yet you often see such self reported data quoted by articles or papers as if it were factual.
Re: Statistical challenges and misreadings of literature create unreplicable science [pdf]
#3On page 11 there is a mention of taking the result of self reporting (surveys) at their word. I’ve wondered about this issue not just in science but other situations. For example political polling, data point in time surveys, census, etc. Without verification, what good is the data? And yet you often see such self reported data quoted by articles or papers as if it were factual.
It is a proxy.
Re: Statistical challenges and misreadings of literature create unreplicable science [pdf]
#4On page 11 there is a mention of taking the result of self reporting (surveys) at their word. I’ve wondered about this issue not just in science but other situations. For example political polling, data point in time surveys, census, etc. Without verification, what good is the data? And yet you often see such self reported data quoted by articles or papers as if it were factual.
For many types of data you'd like, there is no alternative. For example for political polls, especially early ones where you're asking people who they would vote for if so and so were running, what other tool could you use? Or how would you verify the results?
Of course, given the weakness of the data, you have to remember to place very little trust in the results. Even so, "X% of people say they would vote for Y" can be an interesting information in itself, even if those people were lying.
Re: Statistical challenges and misreadings of literature create unreplicable science [pdf]
#5On page 11 there is a mention of taking the result of self reporting (surveys) at their word. I’ve wondered about this issue not just in science but other situations. For example political polling, data point in time surveys, census, etc. Without verification, what good is the data? And yet you often see such self reported data quoted by articles or papers as if it were factual.
For many types of data you'd like, there is no alternative. For example for political polls, especially early ones where you're asking people who they would vote for if so and so were running, what other tool could you use? Or how would you verify the results? Of course, given the weakness of the data, you have to remember to place very little trust in the results. Even so, "X% of people say they would vote for Y" ca…
[deleted]
Re: Statistical challenges and misreadings of literature create unreplicable science [pdf]
#6On page 11 there is a mention of taking the result of self reporting (surveys) at their word. I’ve wondered about this issue not just in science but other situations. For example political polling, data point in time surveys, census, etc. Without verification, what good is the data? And yet you often see such self reported data quoted by articles or papers as if it were factual.
Pollsters know about this, and the rule of thumb is that you can reliably ask people of what they currently do routinely, almost daily. E.g. if you ask them who they voted 10 years ago, they might remember it wrong. If you ask what they're gonna do, this is completely unreliable.
In political polls, this is the only thing you can do. I guess, they have corrections for the real attendance of social/age strata.
In deep interviews, the sociologists are more interested in how people explain their world picture and what they point at, and how react to questions, than to what they tell about themselves.
Re: Statistical challenges and misreadings of literature create unreplicable science [pdf]
#7On page 11 there is a mention of taking the result of self reporting (surveys) at their word. I’ve wondered about this issue not just in science but other situations. For example political polling, data point in time surveys, census, etc. Without verification, what good is the data? And yet you often see such self reported data quoted by articles or papers as if it were factual.
For many types of data you'd like, there is no alternative. For example for political polls, especially early ones where you're asking people who they would vote for if so and so were running, what other tool could you use? Or how would you verify the results? Of course, given the weakness of the data, you have to remember to place very little trust in the results. Even so, "X% of people say they would vote for Y" ca…
Polls may or may not be carried out in a neutral way. A good many are deliberately biased, even more have known flaws that are glossed over. A humorous take on this I love (from the 80s BBC comedy Yes Minister, which is often scarily realistic):
Re: Statistical challenges and misreadings of literature create unreplicable science [pdf]
#8We're increasingly aware today of how the media operates cycles of self-referential and self-justifying citations: a TV show will quote an article that reports "some people" taking an issue, which ends up being a quote from someone interviewed for another newspaper article.. and so on. This "legitimacy laundering" is rampant, and we're now getting towards media literacy levels which expose it for many people.
However, most concerning: this is how academia has always worked. It's the great absurdity of peer review, and of citation. This is how entire fields can sustain themselves with little or no scientific validity (esp. see, psychometrics).
We are no where near the equivalent "academic literacy" for generally informed members of the public to understand this problem. Entire fields can be sustained with zero "empirical pressure" to close down. So long as one can cite another who can cite another... and somewhere some government body will take these citations as prima facie evidence of a research programme, then funding will be given and more papers published.
Re: Statistical challenges and misreadings of literature create unreplicable science [pdf]
#9We're increasingly aware today of how the media operates cycles of self-referential and self-justifying citations: a TV show will quote an article that reports "some people" taking an issue, which ends up being a quote from someone interviewed for another newspaper article.. and so on. This "legitimacy laundering" is rampant, and we're now getting towards media literacy levels which expose it for many people. However…
Economics works exactly like this, but at a fractal level: there are many different subfields living in the grand family of economists which barely communicate with one another, often have dramatically different theories that lead to opposing conclusion, none of which even trying to give an accurate description of the world but instead promote one particular ideological bias.
Every once is a while someone from one chapel is going to actually do science by refuting one other chapel's bullshit, but it's almost never seen as a good thing by anyone, and the Nobel committee sometimes ends up giving the price together to people from different chapels even when their work contradict each other (Fama being empirically being proven wrong by Schiller and both being given the price in 2013 to appease the feud).
Economic behaviors are so complex that studying economics seriously requires gigantic efforts for little results, and people who attempt to do that stand no chance of survival in the publish-or-perish competition against the grandiose bullshitters, and that's how we get where we are.
Re: Statistical challenges and misreadings of literature create unreplicable science [pdf]
#10Earlier quoted context omitted.
For many types of data you'd like, there is no alternative. For example for political polls, especially early ones where you're asking people who they would vote for if so and so were running, what other tool could you use? Or how would you verify the results? Of course, given the weakness of the data, you have to remember to place very little trust in the results. Even so, "X% of people say they would vote for Y" ca…
Polls may or may not be carried out in a neutral way. A good many are deliberately biased, even more have known flaws that are glossed over. A humorous take on this I love (from the 80s BBC comedy Yes Minister, which is often scarily realistic): https://www.youtube.com/watch?v=G0ZZJXw4MTA
Oh, absolutely, even in more subtle, unintentional ways (order of responses, whether you offer a neutral option or not, etc). But that doesn't mean polling is entirely useless, or that you need to verify the responses in the way that the original seemed to suggest.