No evidence for nudging =/= nudging doesn't exist. I'm fairly sure anyone who has done A/B testing at scale has plenty of evidence that nudging works. Perhaps not up to the standard of science, but there are literally people who manipulate choice architecture for a living and I'm fairly convinced a lot of that stuff actually works.
What exactly makes you convinced that it works? To be specific: why wouldn’t there be bias in the A/B testing results, too? There are literally people who give astrological analyses for a living.
No evidence for nudging after adjusting for publication bias
21–30 of 246 posts
Re: No evidence for nudging after adjusting for publication bias
#22What's exactly "nudging" here?. For example it has been shown that for organ donation, if the default is affirmative (opt-in) and you have to opt-out, then organ donors double https://www.science.org/doi/10.1126/science.1091721 , I think this is one of the "nudge" example in pop-science books.
Re: No evidence for nudging after adjusting for publication bias
#23If publication bias is the exclusion of publishing results that doesn’t support your hypothesis, how are they taking that into account? If I’m interpreting this correctly(and I by no means am sure that I am), I infer that they are saying in a fair publishing environment you’d expect to see more results that show less decisive results, therefor the current set of results is likely biased. Couldn’t this bias also happe…
A funnel plot plots effect sizes on the x axis and precision on the y axis. The most precise studies should be tightly grouped around the meta-analytic average effect; the least precise studies should be spread more widely. This forms a triangular, funnel shape. If no publication bias exists, the spread of studies below the magnitude of the average effect should be comparable to the spread of studies above the magnitude of the average effect.
If there is publication bias, then the points that would form the left (without loss of generality; right if negative effect size) portion of the funnel will not be observable.
There are issues with funnel plots and there are other diagnostics but I hope this provides insight into one of the tools used. Notably, as a diagnostic, funnel plots work whether the true effect is positive, negative, or null; they assume only that the underlying assumptions of meta-analysis are true (that studies represent a sample of the same, true underlying effect -- other diagnostics and corrections exist when this is violated)
Re: No evidence for nudging after adjusting for publication bias
#24I agree with other commenters that it's unlikely nudges never have an impact. We should also be wary of high-profile debunkings, now that they're increasingly in fashion due to the replication crisis and the general dour mood. It's easy to p-hack a result into significance, but you can just as easily hack results into insignificance. These days, both findings and debunkings need a skeptical eye.
Re: No evidence for nudging after adjusting for publication bias
#25Earlier quoted context omitted.
What exactly makes you convinced that it works? To be specific: why wouldn’t there be bias in the A/B testing results, too? There are literally people who give astrological analyses for a living.
We are talking about publication bias, where the decision whether to publish something is biased by the outcome of the experiment. I think this doesn't really apply to A/B testing, because people are incentivized pay as much attention to negative results as to positive ones.
Re: No evidence for nudging after adjusting for publication bias
#26No evidence for nudging =/= nudging doesn't exist. I'm fairly sure anyone who has done A/B testing at scale has plenty of evidence that nudging works. Perhaps not up to the standard of science, but there are literally people who manipulate choice architecture for a living and I'm fairly convinced a lot of that stuff actually works.
Re: No evidence for nudging after adjusting for publication bias
#27Re: No evidence for nudging after adjusting for publication bias
#28What's exactly "nudging" here?. For example it has been shown that for organ donation, if the default is affirmative (opt-in) and you have to opt-out, then organ donors double https://www.science.org/doi/10.1126/science.1091721 , I think this is one of the "nudge" example in pop-science books.
Thaler and Sunstein wrote the book on nudges, quite literally. So their definition counts, and it's the one from the article. The opt-in/out decision you mention isn't a nudge in this sense. You're not asked what you prefer, you have to be aware that you can opt-in/out and then actively pursue that option.
Re: No evidence for nudging after adjusting for publication bias
#29Reading this article, it is good that the UK COVID policy wasn't based on behavioural nudging /sarcasm. The UK COVID policy heavily relied on this and one of the unwanted side effects was scaring a certain section of the population into submission. Although that may have been effective during COVID, it made it a lot harder for that segment of society to return back to normal.
Re: No evidence for nudging after adjusting for publication bias
#30No evidence for nudging =/= nudging doesn't exist. I'm fairly sure anyone who has done A/B testing at scale has plenty of evidence that nudging works. Perhaps not up to the standard of science, but there are literally people who manipulate choice architecture for a living and I'm fairly convinced a lot of that stuff actually works.
"... evidence that nudging works. Perhaps not up to the standard of science..." That's pretty close to saying it doesn't work. The point of this meta-study was precisely to show that the evidence claimed to support nudging was probably attributable to random variation + unnatural selection, where the unnatural selection was publication choice: either the researchers who got negative (null) results chose not to bother…
No it's really not.
To say things a different way, I don't think this study will change anything for people actually doing choice architecture in applied settings. They have results that speak for themselves.