Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Scientists Perturbed by Loss of Stat Tool to Sift Research Fudge from Fact

scientificamerican.com

Post reply on HN
Print view View original

31–40 of 40 posts

Re: Scientists Perturbed by Loss of Stat Tool to Sift Research Fudge from Fact

#31

Post by visos »

This is a bad idea. Sure, p-test is pretty flawed, but this is like going without an antivirus because the one you have has bad detection rates.

The researchers are not the only ones who could game the system. A bigger problem is the editorial staff. Replacing an objective test, however bad, with a nonspecific 'case by case' criteria opens the door for nepotism and political agenda pushing. Psychology is an especially dangerous field for this, with the potential to label entire groups of people with opposing views as mentally ill.

The cynic in me sees this as a power-grab.

What they should have done is specify Bayesianism as the new test, period. None of this case-by-case BS.

Re: Scientists Perturbed by Loss of Stat Tool to Sift Research Fudge from Fact

#32

Post by analog31 »

captainmuon wrote: post #22

I find this whole backlash against p-values pretty confusing. That is probably because I come from particle physics, where we also use a lot of statistics, but in subtly different ways. Hypothesis testing is not too hard [1]. You pick a cutoff, say p Since we are a cautious bunch, we actually put the threshold for discovery at 5 sigma - p One other thing that we have to take into account - and many people forget this…

I studied physics too, and am not a statistician.

My way of interpreting an "absurd" threshold such as 5-sigma, is that you can stop using hypothesis testing, and start using other tools for interpreting data.

Another interpretation is that we set these huge limits because, deep in our hearts, we aren't really comfortable with hypothesis testing. There's Rutherford's famous quip: If your experiment needs statistics, then you should have done a better experiment.

I think statistics are actually used pretty sparingly and with caution in physics. There are lots of experiments where the presence or absence of a phenomenon is not the primary question. I didn't report a p-value at all in my thesis project.

Re: Scientists Perturbed by Loss of Stat Tool to Sift Research Fudge from Fact

#33

Post by kremlin »

captainmuon wrote: post #22

I find this whole backlash against p-values pretty confusing. That is probably because I come from particle physics, where we also use a lot of statistics, but in subtly different ways. Hypothesis testing is not too hard [1]. You pick a cutoff, say p Since we are a cautious bunch, we actually put the threshold for discovery at 5 sigma - p One other thing that we have to take into account - and many people forget this…

Your explanation worked really well for me. I now understand what it is bayesians don't like about p value reliance.

Re: Scientists Perturbed by Loss of Stat Tool to Sift Research Fudge from Fact

#34

Post by raverbashing »

captainmuon wrote: post #22

I find this whole backlash against p-values pretty confusing. That is probably because I come from particle physics, where we also use a lot of statistics, but in subtly different ways. Hypothesis testing is not too hard [1]. You pick a cutoff, say p Since we are a cautious bunch, we actually put the threshold for discovery at 5 sigma - p One other thing that we have to take into account - and many people forget this…

Sample size. Also particles are "the same" and much simpler.

Put a particle in a magnetic field and it will behave the same every time (and you can usually do this 1e3, 1e4, 1e5 times as needed). Give 100 people the same drug (ok, 50 of them get the drug, 50 get the placebo) and see each one have a different reaction.

Re: Scientists Perturbed by Loss of Stat Tool to Sift Research Fudge from Fact

#35

Post by cshimmin »

raverbashing wrote: post #34
captainmuon wrote: post #22

I find this whole backlash against p-values pretty confusing. That is probably because I come from particle physics, where we also use a lot of statistics, but in subtly different ways. Hypothesis testing is not too hard [1]. You pick a cutoff, say p Since we are a cautious bunch, we actually put the threshold for discovery at 5 sigma - p One other thing that we have to take into account - and many people forget this…

Sample size. Also particles are "the same" and much simpler. Put a particle in a magnetic field and it will behave the same every time (and you can usually do this 1e3, 1e4, 1e5 times as needed). Give 100 people the same drug (ok, 50 of them get the drug, 50 get the placebo) and see each one have a different reaction.

Actually the whole point of particle physics is that the same thing _doesn't_ happen every time. Quantum field theory allows us to predict probabilities e.g. of a particle decaying to another particle, for a given model. As for doing something 1e4, 1e5 times, at the LHC we collided bunches of 10e10 particles continuously every 50ns for months, and still only produced a few hundred Higgs particles. So these studies of ultra rare processes tend to also have limited statistical power.

Re: Scientists Perturbed by Loss of Stat Tool to Sift Research Fudge from Fact

#36

Post by raverbashing »

cshimmin wrote: post #35
raverbashing wrote: post #34

Earlier quoted context omitted.

Sample size. Also particles are "the same" and much simpler. Put a particle in a magnetic field and it will behave the same every time (and you can usually do this 1e3, 1e4, 1e5 times as needed). Give 100 people the same drug (ok, 50 of them get the drug, 50 get the placebo) and see each one have a different reaction.

Actually the whole point of particle physics is that the same thing _doesn't_ happen every time. Quantum field theory allows us to predict probabilities e.g. of a particle decaying to another particle, for a given model. As for doing something 1e4, 1e5 times, at the LHC we collided bunches of 10e10 particles continuously every 50ns for months, and still only produced a few hundred Higgs particles. So these studies of…

Yes, in quantum physics you have a set of possibilities and their probabilities, and with more experiments you'll produce more Higgs particles, for example

Re: Scientists Perturbed by Loss of Stat Tool to Sift Research Fudge from Fact

#37

Post by vacri »

cshimmin wrote: post #35
raverbashing wrote: post #34

Earlier quoted context omitted.

Sample size. Also particles are "the same" and much simpler. Put a particle in a magnetic field and it will behave the same every time (and you can usually do this 1e3, 1e4, 1e5 times as needed). Give 100 people the same drug (ok, 50 of them get the drug, 50 get the placebo) and see each one have a different reaction.

Actually the whole point of particle physics is that the same thing _doesn't_ happen every time. Quantum field theory allows us to predict probabilities e.g. of a particle decaying to another particle, for a given model. As for doing something 1e4, 1e5 times, at the LHC we collided bunches of 10e10 particles continuously every 50ns for months, and still only produced a few hundred Higgs particles. So these studies of…

It's still considerably less than the variability in sciences based in biology, though.

Re: Scientists Perturbed by Loss of Stat Tool to Sift Research Fudge from Fact

#38

Post by disgruntledphd2 »

yummyfajitas wrote: post #21
protonfish wrote: post #8

Earlier quoted context omitted.

And what should be used as an alternative? There is no reason to believe banning p value would result in improved research quality. Desperate researchers will just find another technique to game.

Instead of blindly applying a tool you don't understand, you'll need to build a statistical model, explain why it's valid, and then construct a meaningful measurement based on it. Then the referee and reader will be required to understand it and will have the ability to critique it.

Which is good, as it will discourage people from submitting there, which means more potential papers for those of us who got on this train a while ago.

Re: Scientists Perturbed by Loss of Stat Tool to Sift Research Fudge from Fact

#39

Post by po »

The video that made it all come together for me was this "Dance of the P-Values" video:

https://www.youtube.com/watch?v=5OL1RqHrZQ8

It does a great job of showing how the same experiment can yield vastly different p-values and why it's not useful for the task it's been given.

Re: Scientists Perturbed by Loss of Stat Tool to Sift Research Fudge from Fact

#40

Post by hudibras »

megablast wrote: post #26
captainmuon wrote: post #22

I find this whole backlash against p-values pretty confusing. That is probably because I come from particle physics, where we also use a lot of statistics, but in subtly different ways. Hypothesis testing is not too hard [1]. You pick a cutoff, say p Since we are a cautious bunch, we actually put the threshold for discovery at 5 sigma - p One other thing that we have to take into account - and many people forget this…

> which sometimes gets us ridiculed by statisticians You get laughed at because you have this huge luxury at dissecting huge amounts of data, running the experiment billions of times. Do you understand that in most disciplines, you don't have that luxury?

To begin with, you'll have to explain to the physicists that there are other disciplines besides physics.
Post reply on HN