Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

No evidence for nudging after adjusting for publication bias

pnas.org

Post reply on HN
Print view View original

81–90 of 246 posts

Re: No evidence for nudging after adjusting for publication bias

#81

Post by mikkergp »

mcswell wrote: post #16
akyu wrote: post #11

No evidence for nudging =/= nudging doesn't exist. I'm fairly sure anyone who has done A/B testing at scale has plenty of evidence that nudging works. Perhaps not up to the standard of science, but there are literally people who manipulate choice architecture for a living and I'm fairly convinced a lot of that stuff actually works.

"... evidence that nudging works. Perhaps not up to the standard of science..." That's pretty close to saying it doesn't work. The point of this meta-study was precisely to show that the evidence claimed to support nudging was probably attributable to random variation + unnatural selection, where the unnatural selection was publication choice: either the researchers who got negative (null) results chose not to bother…

I think what’s not clear is what’s in those papers and what exactly they have to say about nudging and what definition they’re using. It defies credulity to think that changing defaults in software doesn’t change behavior if only because most users aren’t technically savvy enough to change their settings.

On the other hand the dream of nudge theory is something like a study done in the UK that suggests that adding the line “most of your fellow citizens pay their taxes” will increase the likelihood that people pay taxes. This I’d be more likely to believe the benefits are not clear, and more importantly difficult to replicate across time and culture.

It seems that trying to do a meta-analysis on all of nudge theory (or large categories of it) would indeed show know impact. It’s not like you’re testing one thing, you’re comparing well designed programs, with ones that aren’t.

Re: No evidence for nudging after adjusting for publication bias

#82

Post by svnt »

mikkergp wrote: post #63
a_simm wrote: post #37

Wow. As a former social scientist with an axe to grind this hits hard. I like to provide that HN community with some context as to what this means. There are some 300 “research” departments in each of the major social sciences: psychology, sociology, economics and anthropology. If you believe what they say, about half of their mission is teaching and the other half is research. That’s a lot, tens of billions of dolla…

How does a meta analysis of something like this avoid, I don’t know what it would be called but like regression to the mean. A “nudge” isn’t a singular thing, it’s a very diverse process requiring a competent administrator. My gut would say when you averaged all those out, you’d see no effect because your experimenting, some work some don’t work, some backfire. It seems like you’d have to do a meta analysis on a spec…

They aren’t summing effects. An effect is not cancelled by an inverse effect or as you put it, backfire.

The methodology should (I haven’t investigated theirs in detail) not be susceptible to this, and I doubt a mean of effects would make it through peer review for reasons including the ones you’ve mentioned.

Re: No evidence for nudging after adjusting for publication bias

#83

Post by rjmunro »

Reading this, I don't get how you can take all "nudging" and declare "No evidence". Surely "nudging" encompasses a whole range of different actual actions. Some nudges work, some don't. You can't just average across all of them.

I'm probably totally misunderstanding, but it sounds similar to saying "there is no evidence for medicine" because you've averaged all the papers describing medical interventions that work and those that don't.

I thought the point of "nudges" is that they are so cheap to implement you can easily afford to try many. Most won't work, some will.

Re: No evidence for nudging after adjusting for publication bias

#84

Post by BeetleB »

nabla9 wrote: post #38
civilized wrote: post #17

I agree with other commenters that it's unlikely nudges never have an impact. We should also be wary of high-profile debunkings, now that they're increasingly in fashion due to the replication crisis and the general dour mood. It's easy to p-hack a result into significance, but you can just as easily hack results into insignificance. These days, both findings and debunkings need a skeptical eye.

There is no evidence of impact. That's different from the existence of the phenomenon. Same thing happened to Kahneman, Daniel (2011) and his book of Thinking, Fast and Slow. He acknowledges that several pieces of evidence he presents in the book has disappeared and can't be replicated. He still thinks he is right, he just admits that he does not have strong evidence anymore. What is left is a theory with less and le…

> He acknowledges that several pieces of evidence he presents in the book has disappeared and can't be replicated.

He mostly says that about just one chapter. A significant portion of the book is fallacies of basic statistics and logic.

Re: No evidence for nudging after adjusting for publication bias

#85

Post by tgv »

ghaff wrote: post #45
tgv wrote: post #28

Earlier quoted context omitted.

> A nudge, according to Thaler and Sunstein is any form of choice architecture that alters people's behaviour in a predictable way without restricting options or significantly changing their economic incentives. To count as a mere nudge, the intervention must require minimal intervention and must be cheap. Thaler and Sunstein wrote the book on nudges, quite literally. So their definition counts, and it's the one from…

In the case of organ donation, Thaler has written that he actually prefers mandated choice--i.e. there is no default but you have to either opt in or opt out--in this case. [1] But I'm not sure why a system where the government creates a default of either opt-in or opt-out (which you can change) wouldn't be a nudge. [1] https://www.nytimes.com/2009/09/27/business/economy/27view.h... (The issue being that in countries…

The system is that you're by default in some register. The choice has already been made. Many people aren't even aware of it, or only remotely. You have to undertake action to change it. That's not a "choice architecture" in the nudge sense. That would require that you are presented with both options simultaneously, and are forced to choose. A nudge, e.g. on a web form, would then be to have one option already checked.

Re: No evidence for nudging after adjusting for publication bias

#86

Post by dr_dshiv »

MereInterest wrote: post #64
dr_dshiv wrote: post #52

Earlier quoted context omitted.

Seriously, what about that kind of publication bias: A/B tests don’t get published. If you run a useful system where it would be meaningful and interesting to know whether a social science theory actually applied, you might run an A/B test to see if it works. If it works, it is adopted—but it is almost never published. And that is for two reasons: 1. no incentive to publish and 2. major incentive not to publish. #2 i…

The standards of selecting a result of an A/B test are less stringent than those of publication for the advancement of knowledge. For publication, the goal is to determine whether a model is accurate. For A/B testing, the goal is to select the best design/intervention. The difference is that for scientific testing "inconclusive" means that there isn't enough evidence to consider it a solved problem and it should have…

I think you are missing the point. With academic publication bias, sometimes an unbiased coin gets heads 600 times by chance. Those studies get published. But, if you ran the test again, you might only get 525. That study won’t get published.

And, in opposition to your assumption: there is nothing to prevent A/B tests being published with high academic standards— like a low p value and tons of n. In an academic context, that’s just fine— it’s a small but significant effect.

A/B tests are simply controlled experiments—which are the gold standard of scientific evidence generation in psychology. My point is that the main generators of this evidence are only permitted to use this evidence to inform commerce not public knowledge. That is a loss for science and public policy, in my opinion.

Re: No evidence for nudging after adjusting for publication bias

#87

Post by mikkergp »

svnt wrote: post #82
mikkergp wrote: post #63

Earlier quoted context omitted.

How does a meta analysis of something like this avoid, I don’t know what it would be called but like regression to the mean. A “nudge” isn’t a singular thing, it’s a very diverse process requiring a competent administrator. My gut would say when you averaged all those out, you’d see no effect because your experimenting, some work some don’t work, some backfire. It seems like you’d have to do a meta analysis on a spec…

They aren’t summing effects. An effect is not cancelled by an inverse effect or as you put it, backfire. The methodology should (I haven’t investigated theirs in detail) not be susceptible to this, and I doubt a mean of effects would make it through peer review for reasons including the ones you’ve mentioned.

Where are they getting an effect size of .08 if not by mathing a bunch of other effect sizes.

Re: No evidence for nudging after adjusting for publication bias

#88

Post by svnt »

tgv wrote: post #28
lazyant wrote: post #10

What's exactly "nudging" here?. For example it has been shown that for organ donation, if the default is affirmative (opt-in) and you have to opt-out, then organ donors double https://www.science.org/doi/10.1126/science.1091721 , I think this is one of the "nudge" example in pop-science books.

> A nudge, according to Thaler and Sunstein is any form of choice architecture that alters people's behaviour in a predictable way without restricting options or significantly changing their economic incentives. To count as a mere nudge, the intervention must require minimal intervention and must be cheap. Thaler and Sunstein wrote the book on nudges, quite literally. So their definition counts, and it's the one from…

A nudge is naive if not circularly defined then as it presumes at least two permanently distinct classes of humans: informed humans who can architect nudges and learn about them and other humans who must respond in the same way every time and cannot do this meta-learning.

Re: No evidence for nudging after adjusting for publication bias

#89

Post by ghaff »

mcswell wrote: post #12
mikkergp wrote: post #2

If publication bias is the exclusion of publishing results that doesn’t support your hypothesis, how are they taking that into account? If I’m interpreting this correctly(and I by no means am sure that I am), I infer that they are saying in a fair publishing environment you’d expect to see more results that show less decisive results, therefor the current set of results is likely biased. Couldn’t this bias also happe…

"Couldn’t this bias also happen in the other direction?" The general notion is that positive results ("we tried nudging and succeeded in getting people to behave more in X way") are more publishable than negative results ("we tried nudging, and nothing much happened"). It is a common and well-attested problem in many areas of science, but probably particularly in behavioral sciences; I have not heard of cases where p…

There's also an implicit assumption with nudges/defaults that you're nudging people towards a reasonable place for some combination of policy and preference reasons.

But imagine that a company would just as soon not pay out more 401k matching than it has to, so it makes the default zero. (Which of course is often the norm for different reasons.) That's as valid a nudge as anything but we shouldn't be surprised if a lot of people don't go with the default.

We probably also shouldn't be surprised if a lot of people maybe wouldn't go with a maxed out default.

Defaults wouldn't be nearly so powerful if they weren't typically chosen to be fairly reasonable for the average person in the target audience.

Re: No evidence for nudging after adjusting for publication bias

#90

Post by mikkergp »

ghaff wrote: post #89
mcswell wrote: post #12

Earlier quoted context omitted.

"Couldn’t this bias also happen in the other direction?" The general notion is that positive results ("we tried nudging and succeeded in getting people to behave more in X way") are more publishable than negative results ("we tried nudging, and nothing much happened"). It is a common and well-attested problem in many areas of science, but probably particularly in behavioral sciences; I have not heard of cases where p…

There's also an implicit assumption with nudges/defaults that you're nudging people towards a reasonable place for some combination of policy and preference reasons. But imagine that a company would just as soon not pay out more 401k matching than it has to, so it makes the default zero. (Which of course is often the norm for different reasons.) That's as valid a nudge as anything but we shouldn't be surprised if a l…

But I don’t think the goal is “most people can be nudged” so much as “a nudge is a cheap way to increase a behavior by 5-10% which is probably quite significant in policy circles. Low single digit percentage increase in people who pay their taxes would be huge.
Post reply on HN