Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

No evidence for nudging after adjusting for publication bias

pnas.org

Post reply on HN
Print view View original

181–190 of 246 posts

Re: No evidence for nudging after adjusting for publication bias

#181

Post by LegitShady »

_8j50 wrote: post #110
xyzzy123 wrote: post #46

Earlier quoted context omitted.

In the example you cited I don't think that's a nudge? Or is it? I ask because I am sure that changing defaults DEFINITELY works, especially if the user does not have a strong existing preference. You're not really changing user behaviour most of the time, you're changing the outcome of what they're trying to do, which is to reduce their cognitive load by ignoring as much as they possibly can.

401k is a good example, I have had it for my whole career but if there was a form at any point asking me how much of my pay I want to contribute I would have said 0 because I prefer cash at hand than cash some day and all the b.s. health insurance is already taking a lot. But 401k doesn't bother me enough to change the default so I leave it be as some kind of rainy day fund. I didn't like paying the penalty to withdr…

I max out contribution because my employer matches - instant 100% return. If I didn't have that I'm not sure I'd do it.

Re: No evidence for nudging after adjusting for publication bias

#182

Post by tptacek »

rayiner wrote: post #179
tptacek wrote: post #166

Earlier quoted context omitted.

Obama would have appointed Sunstein no matter what book he wrote, because he was one of UChicago's superstar professors (and was at the same time being aggressively courted by schools around the country; he ended up, of course, at Harvard). His appointment was notable not so much for "nudges" as for the fact that Sunstein was probably one of the most conservative appointments Obama made.

Possibly. I recall nudges being a major talking point during the appointment: https://www.politico.com/agenda/story/2015/10/obamas-effort-... .

Your comment makes it seem as if the Nudge work had qualified him for OIRA, when in fact he was probably one of the country's most obvious lay-up candidates for that role. As I think you know, given his own background, it would have been weirder if Obama hadn't found a role in the government for Sunstein.

Re: No evidence for nudging after adjusting for publication bias

#183

Post by whimsicalism »

malisper wrote: post #158

Reading through the article, it seems the actual claim is much, much weaker than the title: > However, all intervention categories and domains apart from “finance” show evidence for heterogeneity, which implies that some nudges might be effective, even when there is evidence against the mean effect So the article is saying when you look at studies of all "nudges" as whole, adjust for publication bias[0], there isn't…

You are misinterpreting what it is saying. It's like multi comparison bias - if you do a bunch of comparisons and find one significant at 0.05, you can go ahead and say that is significant if you had looked at it alone. But if you consider all of the experiments you are doing, you would expect to find one as significant just by random chance even if it isn't significant.

Re: No evidence for nudging after adjusting for publication bias

#184

Post by TameAntelope »

harry8 wrote: post #177
TameAntelope wrote: post #173

Earlier quoted context omitted.

> I don't think it matters if this criticism is right or not. Okay, you and I care about very different things.

>> I don't think it matters if this criticism is right or not. > Okay, you and I care about very different things. Clearly it is being suggested that it doesn't matter with respect to the social studies departments being in dire need of drastic reform. If you don't care about that either direction why are you commenting on this thread? You do actually care one way and you're "point scoring" to further the argument? S…

Hm? I just think it matters if the letter (the original submission) is correct or not.

Re: No evidence for nudging after adjusting for publication bias

#185

Post by Izkata »

richdougherty wrote: post #154
JusticeJuice wrote: post #113

Earlier quoted context omitted.

I was working on a fintech project (gonna be vague as it's not yet released). The legal team told us we couldn't use default choices anywhere, as it could count as giving financial advice. Fair enough. So we designed the onboarding, and there was this choice the user had to make before we could create their account. During testing, we found people were getting really stuck on this choice, to the point of giving up. T…

I think all of those things you listed are kinds of nudges. They are changes in the "choice architecture" that steer the user to an action. In the context of government, a nudge means influencing people to choose something desirable, while still leaving open the option for people to choose what they want (hence preserving liberty). In contrast, a non-nudge solution would be a law or regulation that forces people into…

> that alters people's behavior

I've always understood this part of the description to be more than a single one-off choice, so none of that around the decision point in the financial product would count.

Re: No evidence for nudging after adjusting for publication bias

#186

Post by HardlyCurious »

curiousllama wrote: post #117
a_simm wrote: post #37

Wow. As a former social scientist with an axe to grind this hits hard. I like to provide that HN community with some context as to what this means. There are some 300 “research” departments in each of the major social sciences: psychology, sociology, economics and anthropology. If you believe what they say, about half of their mission is teaching and the other half is research. That’s a lot, tens of billions of dolla…

> This meta-analytic finding turns on the authors’ method for measuring publication bias. Because I accept that, I must believe that this entire body of research, probably the signal behavioral economics work, is essentially worthless! I strongly disagree with this statement, even as someone who believes “nudge” effects are wildly overblown. It means “these studies failed to find evidence” - NOT that there is nothing…

I think the issue here is using science / evidence to push for policy changes when there isn't actually sound science or evidence. That can be done with sound policies that work just as well as it can be done with bad policies. But we should always be concerned when unsound science gets used. It can be used to shut down valid policy debates. And eventually, on a long enough time line, it will get abused by bad actors.

Re: No evidence for nudging after adjusting for publication bias

#187

Post by unityByFreedom »

sacrosancty wrote: post #168
TameAntelope wrote: post #148

Earlier quoted context omitted.

Isn't this just one specific analysis using a very narrow definition of "nudge", one that doesn't even begin to encompass the work being done at those "300 "research" departments"? Further, isn't this using the same data from the original meta analysis that did find "of small to medium size" effect[0]? Why would this, alone, undo decades of research and clear, bright-line conclusions such as the ones cited in my sibl…

I think social scientists have lost the right to the benefit of the doubt. They don't preregister their trials. They don't publish their negative results. They're notoriously bad at statistics, notoriously let their political beliefs distort their conclusions, notoriously scatter their work over thousands of arbitrary, hard-to-compare small-sample-size studies instead of concentrating their resources. I don't think i…

> They don't preregister their trials.

First of all, preregistration is not a requirement for the scientific method, which has functioned well for centuries. That is a recent trend in response to the overflowing amount of haphazardly published science.

Second, it is up to the individual scientist to decide to preregister or not. Some social scientists may preregister.

Third, small sample size may be a fair critique, however that overlooks how difficult it is to collect such data.

You've made a lot of generalizations here that amount to, "social scientists aren't as rigorous as other areas of science, therefore we should only believe studies that disagree with their results". I don't think throwing the baby out with the bath water is helpful. You can take results of studies with small sample sizes with a grain of salt, watch for replication, etc. Lambasting the field as a whole doesn't make sense to me.

Finally, readers should note that this isn't a new argument. People have been making this claim about social science for 120 years, if not longer, but at least since Freud and contemporaries began publishing.

Re: No evidence for nudging after adjusting for publication bias

#188

Post by rayiner »

unityByFreedom wrote: post #187
sacrosancty wrote: post #168

Earlier quoted context omitted.

I think social scientists have lost the right to the benefit of the doubt. They don't preregister their trials. They don't publish their negative results. They're notoriously bad at statistics, notoriously let their political beliefs distort their conclusions, notoriously scatter their work over thousands of arbitrary, hard-to-compare small-sample-size studies instead of concentrating their resources. I don't think i…

> They don't preregister their trials. First of all, preregistration is not a requirement for the scientific method, which has functioned well for centuries. That is a recent trend in response to the overflowing amount of haphazardly published science. Second, it is up to the individual scientist to decide to preregister or not. Some social scientists may preregister. Third, small sample size may be a fair critique,…

> You've made a lot of generalizations here that amount to, "social scientists aren't as rigorous as other areas of science, therefore we should only believe studies that disagree with their results".

I think it’s more “ignore them completely.” It brings “science” into disrepute to let social science associate with the other sciences.

> I don't think throwing the baby out with the bath water is helpful.

There is no baby!

> People have been making this claim about social science for 120 years, if not longer, but at least since Freud and contemporaries began publishing.

Doesn’t that prove the point? It wasn’t science then and isn’t science today.

Re: No evidence for nudging after adjusting for publication bias

#189

Post by zarzavat »

mcswell wrote: post #130
rjmunro wrote: post #83

Reading this, I don't get how you can take all "nudging" and declare "No evidence". Surely "nudging" encompasses a whole range of different actual actions. Some nudges work, some don't. You can't just average across all of them. I'm probably totally misunderstanding, but it sounds similar to saying "there is no evidence for medicine" because you've averaged all the papers describing medical interventions that work an…

There are a number of posts that address this issue, as well as issues raised by responses to this post. I posted somewhere "no true Scotsman"--I'm not claiming my post was enlightening in itself, but the post I was responded to (and the entire thread) was, IMO, enlightening. As for averaging, yes, you can: if a nudge is ineffective, then its result will be random, and a bunch of ineffective nudges will average zero…

> if a nudge is ineffective, then its result will be random

Ineffective and random are different. Ineffective means that the effect size is smaller than required.

For example, if you read "Paracetamol was ineffective for pain relief after surgery" it doesn't necessarily mean that the effect of paracetamol is unpredictable, inconsistent, immeasurable, or that it had no effect or negative effect. You would most likely interpret it to mean that the paracetamol did have an effect but it was insufficient - the patient was still in too much pain.

Similarly, if a nudge intervention was ineffective, it doesn't tell you how it failed, only that it didn't reach the threshold for success. And it certainly doesn't tell you anything about how well an aggregate of some effective and some ineffective results would perform.

Re: No evidence for nudging after adjusting for publication bias

#190

Post by [deleted] »

harry8 wrote: post #178
mcswell wrote: post #125

Earlier quoted context omitted.

No true Scotsman.

Also the other way will come up in these kinds of "arguments" without doubt. "How can these horrible critics say nudges don't work? Have they never been nudged with a loaded revolver? Can they not imagine that working?" Leaving those of us who don't follow the controversy closely in the field and are interested in what has actually been found out and understood about the world with some degree of confidence across ma…

[deleted]
Post reply on HN