Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Is Ego-Depletion a Replicable Effect?

replicationindex.wordpress.com

Post reply on HN
Print view View original

11–20 of 53 posts

Re: Is Ego-Depletion a Replicable Effect?

#11

Post by jldugger »

Houshalter wrote: post #9
joeyrideout wrote: post #2

As a layman when it comes to statistics and academia, this is the first I've heard of "publication bias". It seems like a pretty big problem at first glance, especially if it recurses and calls into question meta-analysis being done to investigate the presence of the bias itself. Sounds like a massive headache. If this problem is as big as it sounds, what are the repercussions?

Oh it's far worse than that. Iirc about a third of studies fail to replicate, and in psychology it might be higher. In fact we even have a control group for science. A fake field of research that spends it's time studying phenomena which can't possibly be real, the paranormal researchers. They produce tons of positive results anyway. See The Control Group is Out of Control : http://slatestarcodex.com/2014/04/28/the-c…

Nosek's finding was that half to two thirds fail to replicate: http://science.sciencemag.org/content/349/6251/aac4716

Re: Is Ego-Depletion a Replicable Effect?

#12

Post by EGreg »

After reading this whole article I do feel a bit tired... does that count as a meta meta study? Just kidding! Let's eat some cookies!

Richard Feynman was right... this whole field of statistical studies should be called something other than science.

http://calteches.library.caltech.edu/51/2/CargoCult.htm

Re: Is Ego-Depletion a Replicable Effect?

#14

Post by Taek »

tylerkahn wrote: post #6
iQuercus wrote: post #3

Earlier quoted context omitted.

Think about it this way: Suppose my theory is that a fair coin is not actually fair but always lands on heads. If I flip a coin one million times and only publish the ~500,000 results that show coin toss coming up as "heads," I could lead people to believe that a fair coin will always land on heads. That's how misleading this bias can be. But anyone can toss a fair coin, so I'd be found out pretty quick. But with mor…

I think what you're describing is just fraud. To take your fair coin example, publication bias is when the only studies that get published are the ones that show an unfairness in the coin – the rest of the studies that simply find 50% heads and 50% tails (aka a fair coin) don't get published because they're not interesting.

Pre-registration solves this issue. If I announce the study before I have the results, we make sure that uninteresting outcomes are published.

Re: Is Ego-Depletion a Replicable Effect?

#15

Post by MaulingMonkey »

tylerkahn wrote: post #6
iQuercus wrote: post #3

Earlier quoted context omitted.

Think about it this way: Suppose my theory is that a fair coin is not actually fair but always lands on heads. If I flip a coin one million times and only publish the ~500,000 results that show coin toss coming up as "heads," I could lead people to believe that a fair coin will always land on heads. That's how misleading this bias can be. But anyone can toss a fair coin, so I'd be found out pretty quick. But with mor…

I think what you're describing is just fraud. To take your fair coin example, publication bias is when the only studies that get published are the ones that show an unfairness in the coin – the rest of the studies that simply find 50% heads and 50% tails (aka a fair coin) don't get published because they're not interesting.

> I think what you're describing is just fraud.

Fraud requires intent. Now, this extreme of bias would typically require said intent, sure. But let's say the experimenter has anterograde amnesia.

They get an idea, flip a coin - oh hey, it's heads! They publish the interesting result. They get an idea, flip a coin - oh, it was tails, nevermind that theory. They get an idea... each time, they don't remember that they've already attempted the theory. No intent, no fraud, just bias.

The good news is this is a pretty unlikely extreme in an individual. Anterograde amnesia is rare. The bad news is nobody can remember that someone else attempted a theory and came up with a negative result that they didn't bother to publish, and even individuals will have a hard time perfectly accounting for and preventing their own bias within a single study. The even worse news is that fraud exists as well.

Re: Is Ego-Depletion a Replicable Effect?

#17

Post by pknight »

Houshalter wrote: post #9
joeyrideout wrote: post #2

As a layman when it comes to statistics and academia, this is the first I've heard of "publication bias". It seems like a pretty big problem at first glance, especially if it recurses and calls into question meta-analysis being done to investigate the presence of the bias itself. Sounds like a massive headache. If this problem is as big as it sounds, what are the repercussions?

Oh it's far worse than that. Iirc about a third of studies fail to replicate, and in psychology it might be higher. In fact we even have a control group for science. A fake field of research that spends it's time studying phenomena which can't possibly be real, the paranormal researchers. They produce tons of positive results anyway. See The Control Group is Out of Control : http://slatestarcodex.com/2014/04/28/the-c…

If your view is a priori that psi effects are impossible all you will see in the studies that 'it must be invalid and flawed or faked'. Yes, this makes people doubt scientific practices in general. You will always feel you are winning that argument, no matter what criteria those scientists meet.

But psi researchers these days jump through far more hoops, run far more replications, spend far more time looking for weaknesses and alternative explanations, garner far more scrutiny and second guessing. To call that a 'fake field of research' when it is more rigorous is disingenuous.

Negative results do get published regularly. And more replications take place. It is quite hilarious really, because it makes people panic about what is wrong with the methods we apply across the board in science. Skeptics just raise the bar abritrarily and shift the goalposts as a means of sweeping things under the rug while calling it scientific rigour.

We have also now gotten to the point that the fact that we commonly have failed replications (across all social science), it means the whole premise of a line of research must be completely flawed. But we should expect failed replications - for one, it's impossible to repeat a set of circumstances exactly. Many replications aren't true replications.

Re: Is Ego-Depletion a Replicable Effect?

#18

Post by hannob »

joeyrideout wrote: post #2

As a layman when it comes to statistics and academia, this is the first I've heard of "publication bias". It seems like a pretty big problem at first glance, especially if it recurses and calls into question meta-analysis being done to investigate the presence of the bias itself. Sounds like a massive headache. If this problem is as big as it sounds, what are the repercussions?

It is an extremely big problem.

In medicine all kinds of things were done to avoid publication bias (most notably trials registers). It still is a big problem, because the countermeasures aren't enforced. In most other areas people don't even try to counter it, which means it's probably much worse and we don't know how bad it is. It's a major reason why so much of science is most likely false.

I suggest you have a look at the work of Ben Goldacre, he's been a major voice in highlighting this issue. He's done a ted talk introducing the problem and later has written a book (Bad Pharma). He's only focussing on medical trials, but you always have to think: "This is happening in all areas of science".

Re: Is Ego-Depletion a Replicable Effect?

#19

Post by fao_ »

Taek wrote: post #14
tylerkahn wrote: post #6

Earlier quoted context omitted.

I think what you're describing is just fraud. To take your fair coin example, publication bias is when the only studies that get published are the ones that show an unfairness in the coin – the rest of the studies that simply find 50% heads and 50% tails (aka a fair coin) don't get published because they're not interesting.

Pre-registration solves this issue. If I announce the study before I have the results, we make sure that uninteresting outcomes are published.

What if funding for a study gets withdrawn because it looks like it'll produce negative/unfavorable results?

Re: Is Ego-Depletion a Replicable Effect?

#20

Post by TeMPOraL »

fao_ wrote: post #19
Taek wrote: post #14

Earlier quoted context omitted.

Pre-registration solves this issue. If I announce the study before I have the results, we make sure that uninteresting outcomes are published.

What if funding for a study gets withdrawn because it looks like it'll produce negative/unfavorable results?

This should be publicly recorded too and maybe eventually we'll have a list of funding bodies that like to minmax profits by damaging research.
Post reply on HN