Live data from Hacker News

No evidence for nudging after adjusting for publication bias

pnas.org

131–140 of 246 posts

Re: No evidence for nudging after adjusting for publication bias

#131
post #37

Wow. As a former social scientist with an axe to grind this hits hard. I like to provide that HN community with some context as to what this means. There are some 300 “research” departments in each of the major social sciences: psychology, sociology, economics and anthropology. If you believe what they say, about half of their mission is teaching and the other half is research. That’s a lot, tens of billions of dolla…

I left psychology around the time that nudging was gaining traction and I haven’t really been following it. But it seems to have a couple of red flags:

First of the definition:

> A nudge is a function of (condition I) any attempt at influencing people’s judgment, choice or behavior in a predictable way (condition a) that is motivated because of cognitive boundaries, biases, routines, and habits in individual and social decision-making posing barriers for people to perform rationally in their own self-declared interests, and which (condition b) works by making use of those boundaries, biases, routines, and habits as integral parts of such attempts.

I find this definition overly permissive and overlay reliant on unnecessary cognitive terms (like judgement and choice; which can be shortened to behavior) or economic terms (like rationality and self interest). As a fan of behaviorism this feels like an attempt to introduce epicycle into a theory that doesn’t need it. This effect—if it exists—can probably be adequately explained with good old classical conditioning and conditional reinforcements. This is the first red flag. That is not to say we can’t look for specific cognitive functions which makes some reinforcement contingencies more effective then others, but nudge feels a bit too general to actually be of any use in a model. It in fact reminds me of Albert Bandura’s theory of self-efficacy, a theory that seems to have reach a dead-end at this point.

The second red flag is the economic presuppositions. When I skim through the literature it feels like they are creating a band-aid on the thoroughly debunked notion of Homo economicus (the believe that human individuals always behave in a rational way optimized for their own self interest). So instead of recognizing the fact human behavior is more complicated, what they try to do is invent a new term to counter-act the instances where biases are “preventing” such a behavior pattern. I find such an effort to be doomed to fail, as—despite the persistence of economists—rational behavior means a different thing for each individual, and there is no “patch” for what economists call “biases”.

Re: No evidence for nudging after adjusting for publication bias

#132

it seems like what they are saying is here is some statistics that suggests that maybe there’s a bunch of missing data that would Show that nudge doesn’t have an effect. Like most science it seems like the conclusion should be, go do more research to see if we’re right, not, you should conclude that nudging doesn’t work.

They're implying that some such research has already been done, but wasn't published because of publication bias against negative results. So the data exists, but is missing from publication. (I suspect it's also possible that more skeptical researchers have been dissuaded from even doing the research, for fear they'd be wasting their time and research budget because after they did the research, it wouldn't get published.)

Re: No evidence for nudging after adjusting for publication bias

#133

Earlier quoted context omitted.

In the example you cited I don't think that's a nudge? Or is it? I ask because I am sure that changing defaults DEFINITELY works, especially if the user does not have a strong existing preference. You're not really changing user behaviour most of the time, you're changing the outcome of what they're trying to do, which is to reduce their cognitive load by ignoring as much as they possibly can.

Some other poster posted that they must have a pretty specific definition of nudge, because it defies credulity that defaults don’t change outcome if only because half the time I don’t read the defaults or know where to find them. I mean I just found out two weeks ago you could change the hacker news banner color. Are you telling me I’m in a statistically insignia can’t minority of hacker news users? Also how many se…

I'm guilty of not reading this paper in any detail but it feels that the default setting "nudge" idea should work as described. So if you e.g. nudge people by setting up a pension plan by default (that they can opt out of) does that seriously fail to cause more people to have a pension? Or is this claiming something else?

Re: No evidence for nudging after adjusting for publication bias

#134
post #37

Wow. As a former social scientist with an axe to grind this hits hard. I like to provide that HN community with some context as to what this means. There are some 300 “research” departments in each of the major social sciences: psychology, sociology, economics and anthropology. If you believe what they say, about half of their mission is teaching and the other half is research. That’s a lot, tens of billions of dolla…

> This meta-analytic finding turns on the authors’ method for measuring publication bias. Because I accept that, I must believe that this entire body of research, probably the signal behavioral economics work, is essentially worthless! I strongly disagree with this statement, even as someone who believes “nudge” effects are wildly overblown. It means “these studies failed to find evidence” - NOT that there is nothing…

Can these effects be explained without inventing a new term? Because if they can then these studies didn’t really find anything did they?

Whenever I see a new term being introduced as an explanation I am hesitant to accept it, as it may turn out to just be explaining the planetary motions with epicycle, when the motions can be easier explained by moving the sun to the center of the solar system instead of the earth.

Re: No evidence for nudging after adjusting for publication bias

#135
post #37

Wow. As a former social scientist with an axe to grind this hits hard. I like to provide that HN community with some context as to what this means. There are some 300 “research” departments in each of the major social sciences: psychology, sociology, economics and anthropology. If you believe what they say, about half of their mission is teaching and the other half is research. That’s a lot, tens of billions of dolla…

> ... the credibility of social science in general is damaged.

Umm, I have some news for you.

Re: No evidence for nudging after adjusting for publication bias

#136
post #123

Earlier quoted context omitted.

> results that speak for themselves. This is exactly how a midwife explained to me why she uses magic crystals. She told me that there's science, and there's results, and that she's seen the crystals work.

I mean, yeah, if she has solid RCT data on thousands to millions of childbirths and has found a statistically significant impact from using the magic crystals, I would support their use. A/B as well as scientific research uses the same basis. The issue is that in fact the midwife will not have such data. The comparison being made is that A/B testing, if run competently, is pretty close to scientific research, in part…

I wonder how many engineers crack open a statistics book to find the correct test versus just plotting box plots and saying "see looks pretty different"

Re: No evidence for nudging after adjusting for publication bias

#137

Earlier quoted context omitted.

In the example you cited I don't think that's a nudge? Or is it? I ask because I am sure that changing defaults DEFINITELY works, especially if the user does not have a strong existing preference. You're not really changing user behaviour most of the time, you're changing the outcome of what they're trying to do, which is to reduce their cognitive load by ignoring as much as they possibly can.

Some other poster posted that they must have a pretty specific definition of nudge, because it defies credulity that defaults don’t change outcome if only because half the time I don’t read the defaults or know where to find them. I mean I just found out two weeks ago you could change the hacker news banner color. Are you telling me I’m in a statistically insignia can’t minority of hacker news users? Also how many se…

The hacker news banner color doesn't matter and few have ever wanted to change it. But your financial position and needs, what % of your salary you can afford to money-hole until retirement, does matter and is pretty individual. It doesn't defy credulity to me that generally people would make a choice about this (when can I retire?), and that the default doesn't influence it.

I grant that it would be surprising if it had no influence at all, but I think the effect is more the social signal that you should want to save the max, that your neighbors probably do (it's the default after all), etc., rather than people completely ignoring/missing it.

Re: No evidence for nudging after adjusting for publication bias

#138

There's a little-known theory (whose name eludes me) that states: any outcome in behaviour is highly dependent on the immediate and unpredictable interplay of various environment variables and their real and perceived effect on the person. this alone cancels the efficacy of any nudges (but some variables may be in the same direction as the nudge--hence the original but misguided nudge research: they were fooled by ra…

Wait until you start reading about genotype by environment interactions and realizing the implications that has on just about everything in biology and society

Re: No evidence for nudging after adjusting for publication bias

#139
post #37

Wow. As a former social scientist with an axe to grind this hits hard. I like to provide that HN community with some context as to what this means. There are some 300 “research” departments in each of the major social sciences: psychology, sociology, economics and anthropology. If you believe what they say, about half of their mission is teaching and the other half is research. That’s a lot, tens of billions of dolla…

Yes, nudging is an extremely well established concept, all the way from theory to policy - there's a Nobel (memorial) prize for the theory, and the UK government explicitly established a 'nudge unit' (the Behavioural Insights Team) to turn it into policy.

Re: No evidence for nudging after adjusting for publication bias

#140

I agree with other commenters that it's unlikely nudges never have an impact. We should also be wary of high-profile debunkings, now that they're increasingly in fashion due to the replication crisis and the general dour mood. It's easy to p-hack a result into significance, but you can just as easily hack results into insignificance. These days, both findings and debunkings need a skeptical eye.

The paper actually explicitly addresses this:

> However, all intervention categories and domains apart from “finance” show evidence for heterogeneity, which implies that some nudges might be effective, even when there is evidence against the mean effect.

Post reply on HN