Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

No evidence for nudging after adjusting for publication bias

pnas.org

Post reply on HN
Print view View original

151–160 of 246 posts

Re: No evidence for nudging after adjusting for publication bias

#151

Post by topaz0 »

rjmunro wrote: post #83

Reading this, I don't get how you can take all "nudging" and declare "No evidence". Surely "nudging" encompasses a whole range of different actual actions. Some nudges work, some don't. You can't just average across all of them. I'm probably totally misunderstanding, but it sounds similar to saying "there is no evidence for medicine" because you've averaged all the papers describing medical interventions that work an…

This is my interpretation as well. Also weird to think about publication bias in this way: "these studies about the effectiveness of snake oil as a drug weren't published, so we must be overestimating how effective drugs are".

The authors do mention that there is likely to be heterogeneity in (real) effect sizes, but somehow still go with this title/abstract.

Maybe there is a valid conclusion that some of the many nudge studies are probably claiming effects that don't exist. That could be interesting in itself. But rejecting the whole field based on this kind of argument seems wrong.

Re: No evidence for nudging after adjusting for publication bias

#152

Post by jolux »

mikkergp wrote: post #58
xyzzy123 wrote: post #46

Earlier quoted context omitted.

In the example you cited I don't think that's a nudge? Or is it? I ask because I am sure that changing defaults DEFINITELY works, especially if the user does not have a strong existing preference. You're not really changing user behaviour most of the time, you're changing the outcome of what they're trying to do, which is to reduce their cognitive load by ignoring as much as they possibly can.

Some other poster posted that they must have a pretty specific definition of nudge, because it defies credulity that defaults don’t change outcome if only because half the time I don’t read the defaults or know where to find them. I mean I just found out two weeks ago you could change the hacker news banner color. Are you telling me I’m in a statistically insignia can’t minority of hacker news users? Also how many se…

You have to have a certain amount of karma to change the banner color.

Re: No evidence for nudging after adjusting for publication bias

#153

Post by mikkergp »

mcswell wrote: post #132
mikkergp wrote: post #108

it seems like what they are saying is here is some statistics that suggests that maybe there’s a bunch of missing data that would Show that nudge doesn’t have an effect. Like most science it seems like the conclusion should be, go do more research to see if we’re right, not, you should conclude that nudging doesn’t work.

They're implying that some such research has already been done, but wasn't published because of publication bias against negative results. So the data exists, but is missing from publication. (I suspect it's also possible that more skeptical researchers have been dissuaded from even doing the research, for fear they'd be wasting their time and research budget because after they did the research, it wouldn't get publi…

Right but they’re implying that statistically, they don’t have a pile of said research on their desk that they’re pointing too.

Re: No evidence for nudging after adjusting for publication bias

#154

Post by richdougherty »

JusticeJuice wrote: post #113
solarkraft wrote: post #73

Earlier quoted context omitted.

I'd love to read more about this. Do you have examples?

I was working on a fintech project (gonna be vague as it's not yet released). The legal team told us we couldn't use default choices anywhere, as it could count as giving financial advice. Fair enough. So we designed the onboarding, and there was this choice the user had to make before we could create their account. During testing, we found people were getting really stuck on this choice, to the point of giving up. T…

I think all of those things you listed are kinds of nudges. They are changes in the "choice architecture" that steer the user to an action.

In the context of government, a nudge means influencing people to choose something desirable, while still leaving open the option for people to choose what they want (hence preserving liberty). In contrast, a non-nudge solution would be a law or regulation that forces people into the desirable option or perhaps a tax on a certain choice.

In your UI, an example of a non-nudge solution would be removing the other options, effectively forcing their decision. Another example of a non-nudge would be charging different fees depending on their decision.

https://en.wikipedia.org/wiki/Nudge_theory#Definition_of_a_n...

"A nudge, as we will use the term, is any aspect of the choice architecture that alters people's behavior in a predictable way without forbidding any options or significantly changing their economic incentives. To count as a mere nudge, the intervention must be easy and cheap to avoid. Nudges are not mandates. Putting fruit at eye level counts as a nudge. Banning junk food does not."

Re: No evidence for nudging after adjusting for publication bias

#155

Post by iambateman »

Keep in mind that the people who publish a correction like this have a lot to gain from getting a lot of attention.

They succeed when lots of people say “what?! This broadly accepted idea is wrong?”

It’s the equivalent of a Buzzfeed headline, even if backed by thoughtful research. The new research may be correct in invalidating the prior experiment’s evidence but the reality is that we all know the “nudge” idea is useful at a practical level. If I ask my two-year old son if he wants milk, the odds that he is drinking milk 5 minutes later skyrocket. The same principle applies to people making all kinds of choices - from buying insurance to picking a University.

Choice architecture matters and we all know it.

Re: No evidence for nudging after adjusting for publication bias

#156

Post by gwd »

iambateman wrote: post #155

Keep in mind that the people who publish a correction like this have a lot to gain from getting a lot of attention. They succeed when lots of people say “what?! This broadly accepted idea is wrong?” It’s the equivalent of a Buzzfeed headline, even if backed by thoughtful research. The new research may be correct in invalidating the prior experiment’s evidence but the reality is that we all know the “nudge” idea is us…

> Keep in mind that the people who publish a correction like this have a lot to gain from getting a lot of attention.

So you're saying that we need to account for publication bias when reading a paper proposing a new way to evaluate publication bias?

Re: No evidence for nudging after adjusting for publication bias

#157

Post by Retric »

a_simm wrote: post #121
curiousllama wrote: post #117

Earlier quoted context omitted.

> This meta-analytic finding turns on the authors’ method for measuring publication bias. Because I accept that, I must believe that this entire body of research, probably the signal behavioral economics work, is essentially worthless! I strongly disagree with this statement, even as someone who believes “nudge” effects are wildly overblown. It means “these studies failed to find evidence” - NOT that there is nothing…

I defend and elucidate. Worthlessness I would define here relative to the amounts of public and of policy attention the nudge findings have received vs net value add modified by these results. Perhaps due to the PR efforts of leading researchers, it was much more than “set defaults intelligently.” The interpretations were more like: we can use social science to shape peoples’ behavior at the margins. Further these ma…

> Taking that a step further …

That step is in no way supported by the evidence provided.

Re: No evidence for nudging after adjusting for publication bias

#158

Post by malisper »

Reading through the article, it seems the actual claim is much, much weaker than the title:

> However, all intervention categories and domains apart from “finance” show evidence for heterogeneity, which implies that some nudges might be effective, even when there is evidence against the mean effect

So the article is saying when you look at studies of all "nudges" as whole, adjust for publication bias[0], there isn't evidence for nudges as a whole. Of course individual nudges could still have a positive impact.

Maybe I'm misinterpreting what it's saying, but as an analogy, that would be like doing a study of all "diets", determining that when you combine data for all studies on diets there isn't a positive effect, then writing an article with the claim "no evidence for diets". There's no way you could reasonably make the claim that no diets work.

[0] If you look at the studies on "nudges", studies with smaller sample sizes detected a larger positive effect size. This is because smaller studies that get a positive effect are more likely to be published than smaller studies with a negative effect. The article uses this to analyze just how strong the publication bias is and adjust for it.

Re: No evidence for nudging after adjusting for publication bias

#159

Post by msrenee »

pessimizer wrote: post #74
akyu wrote: post #30

Earlier quoted context omitted.

>That's pretty close to saying it doesn't work. No it's really not. To say things a different way, I don't think this study will change anything for people actually doing choice architecture in applied settings. They have results that speak for themselves.

> results that speak for themselves. This is exactly how a midwife explained to me why she uses magic crystals. She told me that there's science, and there's results, and that she's seen the crystals work.

Obviously they don't work by magical vibration, but are you sure they don't work at all? If the midwife feels and acts more confident from having that tool or the mother feels more relaxed because she thinks they will make the process easier, then the crystals do, in fact, work. They just don't work through the mechanism those individuals think they do.

Re: No evidence for nudging after adjusting for publication bias

#160

Post by concordDance »

rad88 wrote: post #137
mikkergp wrote: post #58

Earlier quoted context omitted.

Some other poster posted that they must have a pretty specific definition of nudge, because it defies credulity that defaults don’t change outcome if only because half the time I don’t read the defaults or know where to find them. I mean I just found out two weeks ago you could change the hacker news banner color. Are you telling me I’m in a statistically insignia can’t minority of hacker news users? Also how many se…

The hacker news banner color doesn't matter and few have ever wanted to change it. But your financial position and needs, what % of your salary you can afford to money-hole until retirement, does matter and is pretty individual. It doesn't defy credulity to me that generally people would make a choice about this (when can I retire?), and that the default doesn't influence it. I grant that it would be surprising if it…

The default influenced me and pretty much everyone in my company Ive talked to (namely, almost everyone stuck to it). So yes, it defies credulity.
Post reply on HN