Statistical challenges and misreadings of literature create unreplicable science [pdf] • Hacker News Board

Re: Statistical challenges and misreadings of literature create unreplicable science [pdf]

#31

Post by mjburgess » Mon, Nov 11, 2024, 4:22 PM UTC

voidhorse wrote: post #27

mjburgess wrote: post #8

We're increasingly aware today of how the media operates cycles of self-referential and self-justifying citations: a TV show will quote an article that reports "some people" taking an issue, which ends up being a quote from someone interviewed for another newspaper article.. and so on. This "legitimacy laundering" is rampant, and we're now getting towards media literacy levels which expose it for many people. However…

Totally, and I think there is a fundamental, deeper, inherent problem in using statistics to determine how you want to manipulate an given object of study. Researches find that, on average, consumers want X. Companies decide they want to maximize reach and so begin to produce X. Consumers soon have little choice except for X, reaffirming, to future researchers, that consumers want X. This is why I think a plurality o…

I fully agree. How many IQ effects in studied populations are actually created by the use of IQ tests?

Consider how infiltrated IQ-like assessment in throughout society, selecting for doctors, lawyers, postgrads -- military, police, etc. Then consider what data is offered as evidence that IQ 1) exists, and 2) causes observable measures in real-world outcomes. Filtering on the test becomes evidence the test is a measure of anything.

The application of "statistics" I dislike the most is where these feedback cycles exist, and large swathes of academia have some extreme responsibility here.

Whole fields of gene-traits studies were created over decade+ and then disappeared overnight as actual sequencing took place. All the rigour and splendour of "statistics", and then poof when science was done, it disappeared.

Since there are basically no scientific theories of human psychology, society, and the like -- gluing together correlations here should be seen as prima facie absurd. The alternative? Rely on expertise, and build resilience-to-failure into the system and tolerance for higher variability.

Human expertise obtained in domain-specific environments is vastly superior to the species correlations of surveys written by idiots who've never done any actual science.

Re: Statistical challenges and misreadings of literature create unreplicable science [pdf]

#32

Post by kenjackson » Mon, Nov 11, 2024, 4:42 PM UTC

InDubioProRubio wrote: post #29

mjburgess wrote: post #8

We're increasingly aware today of how the media operates cycles of self-referential and self-justifying citations: a TV show will quote an article that reports "some people" taking an issue, which ends up being a quote from someone interviewed for another newspaper article.. and so on. This "legitimacy laundering" is rampant, and we're now getting towards media literacy levels which expose it for many people. However…

Could one create a proof of pseudo-science, by injecting a faked fundamental corner stone paper, that becomes proof by inheritance that a full field is rotten? Also why does this remind me of european politicans, claiming everyone wants to life european lifes, meanwhile whole countries goto war and atrocities without big counter-demonstrations by those western valued citizens .. narrative glider guns going ad absurdu…

> Could one create a proof of pseudo-science, by injecting a faked fundamental corner stone paper, that becomes proof by inheritance that a full field is rotten?

I don't think there's any doubt that pseudoscience exists, even amongst the most optimistic of scientists.

The problem is identifying what is bad science vs what is good. The fact that I can send this from a small phone from a parking lot is proof that someone did good science at some point in time. Or that I've seen therapies like CBT turn someone who was struggling mentally on a daily basis to thrive, or that I've seen valve replacement surgery give someone years of great life after a "six months to live diagnosis" -- all show that there is good science.

I think we almost need a "discipline" of people who validate scientific results, and people should be held accountable for results that validate or don't.

Peer review is great. It's not a farce (I've gotten some incredible feedback on papers from it), but it is also extremely limited by design. We need more.

Re: Statistical challenges and misreadings of literature create unreplicable science [pdf]

#33

Post by brabel » Mon, Nov 11, 2024, 4:56 PM UTC

mjburgess wrote: post #8

We're increasingly aware today of how the media operates cycles of self-referential and self-justifying citations: a TV show will quote an article that reports "some people" taking an issue, which ends up being a quote from someone interviewed for another newspaper article.. and so on. This "legitimacy laundering" is rampant, and we're now getting towards media literacy levels which expose it for many people. However…

Great video about that by Kurzgesagt: https://www.youtube.com/watch?v=bgo7rm5Maqg&ab_channel=Kurzg...

They show how deep they had to go to find the original source of the claim that a single human's blood vessels, if lined up, would stretch 100,000km... and how that was quoted by so many that no one really know where the claim came from. And of course, it was wrong.

Re: Statistical challenges and misreadings of literature create unreplicable science [pdf]

#34

Post by jjk166 » Mon, Nov 11, 2024, 6:09 PM UTC

InDubioProRubio wrote: post #29

mjburgess wrote: post #8

We're increasingly aware today of how the media operates cycles of self-referential and self-justifying citations: a TV show will quote an article that reports "some people" taking an issue, which ends up being a quote from someone interviewed for another newspaper article.. and so on. This "legitimacy laundering" is rampant, and we're now getting towards media literacy levels which expose it for many people. However…

Could one create a proof of pseudo-science, by injecting a faked fundamental corner stone paper, that becomes proof by inheritance that a full field is rotten? Also why does this remind me of european politicans, claiming everyone wants to life european lifes, meanwhile whole countries goto war and atrocities without big counter-demonstrations by those western valued citizens .. narrative glider guns going ad absurdu…

Even a broken clock is right twice a day. I can publish a paper with results I think are wrong, but it's entirely possible that follow up studies confirm it was accidentally correct. While this is improbable in general, if we restrict ourselves to publishing claims that are sufficiently plausible that people in the field would accept them as a corner stone paper, then there is a very decent chance it sounds plausible because it's true. Even if I test the claim myself before publishing to confirm it's false, I may have made an error.

Possibly repeatedly publishing bogus papers in a certain manner might be able to confidently weed out poor academic hygeine, but it's not a trivial thing.

Re: Statistical challenges and misreadings of literature create unreplicable science [pdf]

#35

Post by [deleted] » Mon, Nov 11, 2024, 6:11 PM UTC

mjburgess wrote: post #8

We're increasingly aware today of how the media operates cycles of self-referential and self-justifying citations: a TV show will quote an article that reports "some people" taking an issue, which ends up being a quote from someone interviewed for another newspaper article.. and so on. This "legitimacy laundering" is rampant, and we're now getting towards media literacy levels which expose it for many people. However…

[deleted]

Re: Statistical challenges and misreadings of literature create unreplicable science [pdf]

#36

Post by naasking » Mon, Nov 11, 2024, 6:12 PM UTC

brabel wrote: post #33

mjburgess wrote: post #8

We're increasingly aware today of how the media operates cycles of self-referential and self-justifying citations: a TV show will quote an article that reports "some people" taking an issue, which ends up being a quote from someone interviewed for another newspaper article.. and so on. This "legitimacy laundering" is rampant, and we're now getting towards media literacy levels which expose it for many people. However…

Great video about that by Kurzgesagt: https://www.youtube.com/watch?v=bgo7rm5Maqg&ab_channel=Kurzg... They show how deep they had to go to find the original source of the claim that a single human's blood vessels, if lined up, would stretch 100,000km... and how that was quoted by so many that no one really know where the claim came from. And of course, it was wrong.

Damn it, I had just posted this video too. Off to delete it.

Re: Statistical challenges and misreadings of literature create unreplicable science [pdf]

#37

Post by naasking » Mon, Nov 11, 2024, 6:17 PM UTC

mjburgess wrote: post #26

light_hue_1 wrote: post #24

Earlier quoted context omitted.

And my PhD and dayjob is doing research on this. If a student told me they had this view of what ML is, I would tell them that we've failed to educate them. The thought that physics doesn't care about performance or approximation is silly. Just look at AlphaFold. Heck, I talk to climatologists and material scientists that want the equivalent all the time. Prediction is the heart of all science. Whether we're talking…

Prediction is not the heart of science, this is early 20th C. mumbojumbo and humean nonesense that gets repeated by curve-fitters because it's all they do. Explanation is the heart of science, not prediction. All predictions newton would have made of the orbits of the planets would have been wrong (and so on). And this goes for the vast majority of textbooks physics when its applied to very many ordinary situations:…

Prediction and explanation are basically equivalent IMO. A predictive model entails an explanation, and an explanation entails a predictive model. Predictive models that are more accurate are more accurate explanations, and predictive models that are more precise are more precise explanations, and vice versa. They are not that distinct.

Re: Statistical challenges and misreadings of literature create unreplicable science [pdf]

#38

Post by mjburgess » Mon, Nov 11, 2024, 6:22 PM UTC

naasking wrote: post #37

mjburgess wrote: post #26

Earlier quoted context omitted.

Prediction is not the heart of science, this is early 20th C. mumbojumbo and humean nonesense that gets repeated by curve-fitters because it's all they do. Explanation is the heart of science, not prediction. All predictions newton would have made of the orbits of the planets would have been wrong (and so on). And this goes for the vast majority of textbooks physics when its applied to very many ordinary situations:…

Prediction and explanation are basically equivalent IMO. A predictive model entails an explanation, and an explanation entails a predictive model. Predictive models that are more accurate are more accurate explanations, and predictive models that are more precise are more precise explanations, and vice versa. They are not that distinct.

They are highly distinct.

Compare reporting the temperature tomorrow as a mode of all temperatures in November at your location, with a climate & weather simulation involving: cloud layers, the ocean, etc.

The former is likely to be vastly more predictively accurate than the latter, but explains nothing.

Explanatory models are often less predictively accurate than these (weakly inductive) predictive models. Their purpose is to tell us how reality works, and that provides some insight as to when we can adopt merely predictive approaches. This is because merely predictive models capture accidental features of measurement which hold up for awhile in some environments, that we wouldn't wish to explain.

Without explanatory insight we find merely predictive models catastrophically collapse, and are otherwise, highly fragile. Eg., consider the performance of "predict the mode" in a snow storm.

If you want a midly formal analysis of the difference: explanatory models quantify over causal properties of reality, describe their relationship, and provide necessary inferential methods for deducing conclusions from models. They permit arbitrary simulation across all relevant measuring systems.

Merely predictive models quantify over historical measurements, assume similarity conditions across them, and assume future similarity to past cases. They provide only extremely weak inferential grounds for any inference. They can offer only repetition of one kind of measure, and cannot simulate the state of other relevant measuring devices or in different environments of measurement.

Explanatory models describe reality. Predictive models describe the measurement device you happened to use, in the environment you specifically used it in.

Re: Statistical challenges and misreadings of literature create unreplicable science [pdf]

#39

Post by jjk166 » Mon, Nov 11, 2024, 6:25 PM UTC

blackeyeblitzar wrote: post #2

On page 11 there is a mention of taking the result of self reporting (surveys) at their word. I’ve wondered about this issue not just in science but other situations. For example political polling, data point in time surveys, census, etc. Without verification, what good is the data? And yet you often see such self reported data quoted by articles or papers as if it were factual.

There are multiple levels to data. If 57% of respondents to a self reported survey say they are going to do X, it doesn't mean that 57% of people are actually going to do X. But if you do the same survey again and now only 43% of respondents say they will do X, then that is clear evidence something has clearly changed, even if you don't know exactly what effect that something will have on X. That is very useful.

The problem is when people only look at self reporting. For example if in the previous scenario 57% responded X but only 54% actually did X, then someone might naively assume that after 43% respond X that 40% will actually do X. Or just as naively they could say 54% will still do X. Or they could apply any of an infinite number of other models. There exists a model that will spit out any given answer for any given input, so without that followup work to actually verify and understand the underlying mechanism, models are worthless.

Re: Statistical challenges and misreadings of literature create unreplicable science [pdf]

#40

Post by mrguyorama » Mon, Nov 11, 2024, 6:46 PM UTC

pacman001 wrote: post #14

mjburgess wrote: post #8

We're increasingly aware today of how the media operates cycles of self-referential and self-justifying citations: a TV show will quote an article that reports "some people" taking an issue, which ends up being a quote from someone interviewed for another newspaper article.. and so on. This "legitimacy laundering" is rampant, and we're now getting towards media literacy levels which expose it for many people. However…

Every single time (some 4 times in my past life) when I’m familiar with/close to the background story of something that played out in the news I come to the same conclusion: news doesn’t report the facts. I’m in Western Europe, btw. I advise to run this experiment yourself.

No journalist will ever have the expertise required to accurately report scientific results.

Instead we need to help average people understand that when the news says "Scientists say drinking red wine is healthy!" that No actual scientist ever said that!

Instead, the journalist writing the segment cribbed notes from the University's PR page about the study, which was also written by someone with zero science background, and in fact almost always has a marketing background.

Oh, and those PR releases are outright stating false things that the actual scientific paper doesn't even discuss like half the time.

And now you have decades of people insisting that nutrition science is awful, even though nobody in academia or science is saying any of the things the average person thinks they have.