Live data from Hacker News

Chart of the Decade: Why You Shouldn’t Trust Every Scientific Study You See

motherjones.com

51–60 of 102 posts

Re: Chart of the Decade: Why You Shouldn’t Trust Every Scientific Study You See

#51
post #49

Is it possible that prior to 2000, the studies that found a null result, were simply not published? Hence why they are underrepresented in that chart? My understanding is that it's very hard to get null results published in scientific journals, though I'm not sure if that applies to clinical studies as well.

[deleted]

Re: Chart of the Decade: Why You Shouldn’t Trust Every Scientific Study You See

#52
post #41

Earlier quoted context omitted.

Hardly equivalent "for all intents and purposes." The purposes of the layman are nothing like those of the research community.

If you put it in the public domain it is your responsibility to ensure it is digestible by the audience you publish it to. So yes... public posted information should be layman ready

Do you apply this reasoning to articles or comments discussing kernel development? Some topics have higher irreducible complexity than others.

More importantly, I can make whatever I want public, and I can do so with whatever audience I have in mind.

Re: Chart of the Decade: Why You Shouldn’t Trust Every Scientific Study You See

#53

I 100% agree with the headline; especially for research papers even more than clinical trials. However, I'm a bit puzzled by the weird direction the journalist ran with this, which is straight to his preconceived notions that are not that supported by the data he's looking at. But there's a bit more to this than just that one chart. In addition to self-correction (e.g. beginning to require pre-registration of trials)…

> With the advances in treatments for cardiovascular disease couldn't the problem just be harder? I'm pretty sure we have no significant advances in the treatment of cardiovascular disease. And given the alarmingly climbing rates of chronic diseases, e.g. cardiovascular disease, diabetes, obesity, cancer, I'm also pretty sure that the health care industry has done more harm than good when it comes to chronic diseases…

>And given the alarmingly climbing rates of chronic diseases, e.g. cardiovascular disease, diabetes, obesity, cancer, I'm also pretty sure that the health care industry has done more harm than good when it comes to chronic diseases.

I mean, can't this be attributed to other things too? (The way people live has changed in the past 20 years...)

I'd rather see some studies to show if/why those things have spiked rather than disavowing attempts to help it.

What else are we supposed to do?

Re: Chart of the Decade: Why You Shouldn’t Trust Every Scientific Study You See

#54
post #48

> Then, in 2000, the rules changed. Researchers were required before the study started to say what they were looking for. They couldn’t just mine the data afterward looking for anything that happened to be positive. They had to report the results they said they were going to report. And guess what? Out of 21 studies, only two showed significant benefits. Why is this considered good? Isn't this just a counterproductiv…

"Significant" here doesn't mean what you think it means, but rather it has a technical meaning. Basically, it means that if the intervention has absolutely no effect, then there's less than 5% probability that measured "benefit" is just a random fluke -- thus, since you actually measured benefit, you are now left wondering whether the intervention actually has an effect, or you just happen to be in 5% of possible uni…

> since you actually measured benefit, you are now left wondering whether the intervention actually has an effect, or you just happen to be in 5% of possible universes where the randomness just happened to align this way.

That still seems a good first step. It may also be curious to find out what were the things that have actually lead to the effect if not the one we were checking. E.g. it may happen that it the substance researched only affects subjects with particular features or that something else they were doing has produced the result on itself and knowing any of these still seems useful (some really important discoveries were made this way AFAIK).

Re: Chart of the Decade: Why You Shouldn’t Trust Every Scientific Study You See

#56

I 100% agree with the headline; especially for research papers even more than clinical trials. However, I'm a bit puzzled by the weird direction the journalist ran with this, which is straight to his preconceived notions that are not that supported by the data he's looking at. But there's a bit more to this than just that one chart. In addition to self-correction (e.g. beginning to require pre-registration of trials)…

I would say that not all research needs to be held to the pre-registering standard of clinical trials. Some research is more exploratory, and should be interpreted that way.

Exploratory work is extremely valuable, but less trustworthy ipso facto.

Re: Chart of the Decade: Why You Shouldn’t Trust Every Scientific Study You See

#57

I 100% agree with the headline; especially for research papers even more than clinical trials. However, I'm a bit puzzled by the weird direction the journalist ran with this, which is straight to his preconceived notions that are not that supported by the data he's looking at. But there's a bit more to this than just that one chart. In addition to self-correction (e.g. beginning to require pre-registration of trials)…

> Is it not possible that the low-hanging fruit had been found earlier, in the 1970s-1990s, and the problem got harder? The problem of "problems getting harder" is a continuous phenomenon. Why would that be the case suddenly after 2000, and not before? > However, I'm a bit puzzled by the weird direction the journalist ran with this, which is straight to his preconceived notions that are not that supported by the data…

> The problem of "problems getting harder" is a continuous phenomenon. Why would that be the case suddenly after 2000, and not before?

Well, there was that whole dotcom boom and a lot of things changed for computers & the internet which led to researchers being able to share more information, use more powerful computer techniques, etc.

Re: Chart of the Decade: Why You Shouldn’t Trust Every Scientific Study You See

#58
post #52
post #41

Earlier quoted context omitted.

If you put it in the public domain it is your responsibility to ensure it is digestible by the audience you publish it to. So yes... public posted information should be layman ready

Do you apply this reasoning to articles or comments discussing kernel development? Some topics have higher irreducible complexity than others. More importantly, I can make whatever I want public, and I can do so with whatever audience I have in mind.

Interesting.

So inciting panic online is fine but not in a theater ?

Re: Chart of the Decade: Why You Shouldn’t Trust Every Scientific Study You See

#59

I 100% agree with the headline; especially for research papers even more than clinical trials. However, I'm a bit puzzled by the weird direction the journalist ran with this, which is straight to his preconceived notions that are not that supported by the data he's looking at. But there's a bit more to this than just that one chart. In addition to self-correction (e.g. beginning to require pre-registration of trials)…

> With the advances in treatments for cardiovascular disease couldn't the problem just be harder? I'm pretty sure we have no significant advances in the treatment of cardiovascular disease. And given the alarmingly climbing rates of chronic diseases, e.g. cardiovascular disease, diabetes, obesity, cancer, I'm also pretty sure that the health care industry has done more harm than good when it comes to chronic diseases…

Definitely not accusing you of being an anti-vaxxer, but I would love to hear your sources for some of your statements, because they contradict the data that I've seen. Searching now again, everything I'm finding is showing really big improvements in mortality from cardiovascular disease.

Recent US outcomes: https://jamanetwork.com/journals/jamacardiology/fullarticle/...

US & New York State outcomes 1980-2007, Figure 2: https://www.health.ny.gov/diseases/cardiovascular/heart_dise...

I haven't hit the major mortality sources, but these bits match everything else I've heard.

I'd also like to know the basis for asserting that the health care industry doing more harm than good.

Re: Chart of the Decade: Why You Shouldn’t Trust Every Scientific Study You See

#60
I'd say a good rule of thumb is that the smaller the number of observations, the less you should trust any research that hasn't been reproduced. And when it comes to publication of false positive results, always remember your XKCD: https://xkcd.com/882/
Post reply on HN