Sometimes the implementation of "nudges" is so obvious it has the opposite of the desired effect.
No evidence for nudging after adjusting for publication bias
61–70 of 246 posts
Re: No evidence for nudging after adjusting for publication bias
#62Wow. As a former social scientist with an axe to grind this hits hard. I like to provide that HN community with some context as to what this means. There are some 300 “research” departments in each of the major social sciences: psychology, sociology, economics and anthropology. If you believe what they say, about half of their mission is teaching and the other half is research. That’s a lot, tens of billions of dolla…
> Thus, all that effort has not only been wasted but the credibility of social science in general is damaged.
I don't think that's entirely true If anything this just highlights how complex behavioural science really is, as they're dealing with surprisingly complex humans and their surprisingly complex lives. Behavioural science is a young field.
Re: No evidence for nudging after adjusting for publication bias
#63Wow. As a former social scientist with an axe to grind this hits hard. I like to provide that HN community with some context as to what this means. There are some 300 “research” departments in each of the major social sciences: psychology, sociology, economics and anthropology. If you believe what they say, about half of their mission is teaching and the other half is research. That’s a lot, tens of billions of dolla…
How does a meta analysis of something like this avoid, I don’t know what it would be called but like regression to the mean. A “nudge” isn’t a singular thing, it’s a very diverse process requiring a competent administrator. My gut would say when you averaged all those out, you’d see no effect because your experimenting, some work some don’t work, some backfire. It seems like you’d have to do a meta analysis on a specific nudge, not on groups of nudges.
Re: No evidence for nudging after adjusting for publication bias
#64No evidence for nudging =/= nudging doesn't exist. I'm fairly sure anyone who has done A/B testing at scale has plenty of evidence that nudging works. Perhaps not up to the standard of science, but there are literally people who manipulate choice architecture for a living and I'm fairly convinced a lot of that stuff actually works.
Seriously, what about that kind of publication bias: A/B tests don’t get published. If you run a useful system where it would be meaningful and interesting to know whether a social science theory actually applied, you might run an A/B test to see if it works. If it works, it is adopted—but it is almost never published. And that is for two reasons: 1. no incentive to publish and 2. major incentive not to publish. #2 i…
The standards of selecting a result of an A/B test are less stringent than those of publication for the advancement of knowledge. For publication, the goal is to determine whether a model is accurate. For A/B testing, the goal is to select the best design/intervention. The difference is that for scientific testing "inconclusive" means that there isn't enough evidence to consider it a solved problem and it should have more research, while in A/B testing "inconclusive" means that any effect is small so you should pick an option and move on.
As an example, suppose I flip a coin 1000 times and get heads 525 times. The 95% confidence interval for the probability of heads is [0.494, 0.556], so from a scientific standpoint I cannot conclude that the coin is biased. If, however, I am performing an A/B test, I would conclude that I'll bet on heads, because it is at worst equivalent to tails.
Re: No evidence for nudging after adjusting for publication bias
#65Earlier quoted context omitted.
"... evidence that nudging works. Perhaps not up to the standard of science..." That's pretty close to saying it doesn't work. The point of this meta-study was precisely to show that the evidence claimed to support nudging was probably attributable to random variation + unnatural selection, where the unnatural selection was publication choice: either the researchers who got negative (null) results chose not to bother…
>That's pretty close to saying it doesn't work. No it's really not. To say things a different way, I don't think this study will change anything for people actually doing choice architecture in applied settings. They have results that speak for themselves.
"I don't think this study will change anything for people actually doing choice architecture in applied settings." Probably true, but then evidence that horoscopes etc. don't work, doesn't prevent people from drawing horoscopes, or other people from relying on their horoscope to plan out their day.
"They have results that speak for themselves." Let me put my point differently. Suppose that nudges don't have any effect at all (null hypothesis). More concretely--and just to take a random number--suppose that 50% of the time when a nudge is used, the nudgees happen to behave in the direction that the nudge was intended to move them, and 50% of the time they don't move, or they move in the opposite direction. And suppose there are a number of nudgers, maybe 100. Then some nudgers will get better than random results, while others will get no result, or negative results. The former nudgers will have results that appear to speak for themselves, even if the nudges actually have no effect whatsoever.
This is the same as asking if a fair coin is tossed ten times, what is the probability that you'll get at least 7 heads. The probability of such a number of heads in a single run is ~17%. So 17% of those nudgers could be getting apparently significant results, even if their results are actually random.
Re: No evidence for nudging after adjusting for publication bias
#66I agree with other commenters that it's unlikely nudges never have an impact. We should also be wary of high-profile debunkings, now that they're increasingly in fashion due to the replication crisis and the general dour mood. It's easy to p-hack a result into significance, but you can just as easily hack results into insignificance. These days, both findings and debunkings need a skeptical eye.
I'm a UI designer, and my experience of implementing 'nudges' is that sometimes they work, and sometimes they don't.
The reality is that the way people make decisions is stupidly complex, because people have stupidly complex lives. Some tweak will work great for one project, and do nothing on the next one. It's hard to even say if it was the nudge that worked the first time.
I really view nudge theory as one of many ideas of things you can try, a tool in a toolkit. But the only tool I really feel confident works is the design-test-iterate loop.
Re: No evidence for nudging after adjusting for publication bias
#67Wow. As a former social scientist with an axe to grind this hits hard. I like to provide that HN community with some context as to what this means. There are some 300 “research” departments in each of the major social sciences: psychology, sociology, economics and anthropology. If you believe what they say, about half of their mission is teaching and the other half is research. That’s a lot, tens of billions of dolla…
No offense, but the research in social sciences has very little credibility. From the reproduction crisis to the blatant inability to research things that go against what the professors want, it’s just untrustworthy. Academia needs to fix itself.
Re: No evidence for nudging after adjusting for publication bias
#68There's a lot of confusion here about what this article is talking about. Nudges aren't just defaults. We've known for over a century that people are influenced by defaults. Nudges also aren't anchoring, where choices influence one another. Kahneman & Tversky won a Nobel prize for that and other behavioral economics ideas a decade before the idea of nudges. Nudges are a bigger idea that many small changes lead to hug…
I'm not a social scientist, so please help me understand this. The way you have defined nudge it seems like a very broad category. Some passive nudge (defaults) and some active (red lights) and a whole lot others.
If a category has such a broad number of phenomenon then shouldn't we be analysing individual phenomenon instead of the category as a whole? For example; defaults may work and red-light thing may not work. Why place them both under the same bucket at all? Why not study them in isolation?
Re: No evidence for nudging after adjusting for publication bias
#69The linked study (and the Merten's study it's built ontop of) classifies defaults as "structural" interventions. In the linked meta-analysis, after stripping out estimates of publication basis, structural interventions have the most "robust" evidence left (95% CI of 0.00-0.43), and as their paper text says "whereas the evidence is undecided for “structure” interventions". Other structural interventions include, making it easier to select the desired outcome (or make it harder to switch away from desired out come), changing the range of options to facilitate evaluation, or trying to compensate for biases and loss aversion in choice structure. As you can see, this is a broad range of interventions.
A little bit further they say "However, all intervention categories and domains apart from “finance” show evidence for heterogeneity, which implies that some nudges might be effective, even when there is evidence against the mean effect", which makes sense. People understand stakes generally, and will likely apply different care/effort in different context, modifying the context specific effect of any given intervention.
I think the paper makes a reasonable argument:
1. There is significant publication bias in nudging studies 2. The effect of providing additional information at time of selection, or providing reminders/affirmations for self control is basically non-existent 3. The effect of modifying choice structure is indecisive. Likely we'll find that some structural modifications have strong effects in some context, but others have little or no effect is other context.
Re: No evidence for nudging after adjusting for publication bias
#70Earlier quoted context omitted.
Some other poster posted that they must have a pretty specific definition of nudge, because it defies credulity that defaults don’t change outcome if only because half the time I don’t read the defaults or know where to find them. I mean I just found out two weeks ago you could change the hacker news banner color. Are you telling me I’m in a statistically insignia can’t minority of hacker news users? Also how many se…
If defaults don't work then Google wasted 15 billion dollars last year paying Apple to be in the search bar... I guess there must be further detail in the paper and I will have to read it to understand the nuance.
Also fair to recognise google also pays Apple to not make or promote a competitor that may offer far more competition.
Id say this is a large part of the reason Gmail, Android and Chrome exist.