Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

No evidence for nudging after adjusting for publication bias

pnas.org

Post reply on HN
Print view View original

201–210 of 246 posts

Re: No evidence for nudging after adjusting for publication bias

#201

Post by Schroedingersat »

unityByFreedom wrote: post #193
rayiner wrote: post #188

Earlier quoted context omitted.

> You've made a lot of generalizations here that amount to, "social scientists aren't as rigorous as other areas of science, therefore we should only believe studies that disagree with their results". I think it’s more “ignore them completely.” It brings “science” into disrepute to let social science associate with the other sciences. > I don't think throwing the baby out with the bath water is helpful. There is no b…

> It brings “science” into disrepute to let social science associate with the other sciences. So you just want it renamed to "social studies" or what? What is your proposal, that nobody research this topic, or that they be separated in journals etc? I doubt that will have much impact on whether it makes the news. If you want that to change, you may need to get yourself onto the board of a journal you care about. > Th…

> So you just want it renamed to "social studies" or what? What is your proposal, that nobody research this topic, or that they be separated in journals etc? I doubt that will have much impact on whether it makes the news.

Or maybe people just want to shine light on the fact that social science is harder than other science for a bunch of different reasons, bring social scientists' attention towards the tools that help mitigate this, and bring the journals that seek profit over reliable results into disrepute?

Most of the papers published in current social science journals are not science, and this has been a problem for those 120 years precisely because the techniques used in chemistry or physics are inadequate for the problem domain, so applying them blindly does not produce scientific outcomes.

Re: No evidence for nudging after adjusting for publication bias

#202

Post by JusticeJuice »

richdougherty wrote: post #154
JusticeJuice wrote: post #113

Earlier quoted context omitted.

I was working on a fintech project (gonna be vague as it's not yet released). The legal team told us we couldn't use default choices anywhere, as it could count as giving financial advice. Fair enough. So we designed the onboarding, and there was this choice the user had to make before we could create their account. During testing, we found people were getting really stuck on this choice, to the point of giving up. T…

I think all of those things you listed are kinds of nudges. They are changes in the "choice architecture" that steer the user to an action. In the context of government, a nudge means influencing people to choose something desirable, while still leaving open the option for people to choose what they want (hence preserving liberty). In contrast, a non-nudge solution would be a law or regulation that forces people into…

A nudge has to push the user towards one certain decision over another, that's the whole point. It's opinionated. The factors I listed aren't inherently opinionated, we could've tried to improve them without pushing the user to a specific choice:

E.g. speed: We could've removed earlier parts of the onboarding to make the overall experience less long, or compacted the UI so it was visually easier to skim the choices.

Expertise: we could've assured the user before the choice, that all options were good cause we're the experts and that we would've give you a bad option - so don't agonise.

Cognitive load: We could've reduced the info we showed about each option, or hidden it away behind a modal, or re-written it in plain english. The legal team told us we had to use the legal descriptions of the choices, which included technical language.

Confusion: We could've made an visualisation of the impact of their choice, that changed as they swapped between each option - showing them them a more tangible outcome of their choice. It was a complicated concept to get, so the addition of a visual aid instead of just written descriptions might've helped.

To be clear - I'd be surprised if these things would've worked, and I'm certain setting a default made a difference. The point I'm making is that I don't know for sure how much of a difference. The change to implement the default, by my eyes, also improved the overall design in these other ways as well. We didn't isolate it down to exactly what made the improvement, we were just happy it happened.

The point I'm making is you could quickly skim read this story of a team stuck on a problem, who after implementing defaults found their conversion rates jumped 11x holy shiiiiiiii- and it sounds like it's all thanks to nudge theory. It's exactly like a case study you'd see in a co-design agency's portfolio.

But in the actual real messy world of designing interfaces, it's just always a bit more complicated than that. No change is truly isolated, tested in a controlled, academic fashion. You just design your best shot each time and see what works. Because of this, it's hard to truly definitively say an improvement was because of a nudge. Best I can do is, "I mean probably" haha.

Re: No evidence for nudging after adjusting for publication bias

#204

Post by enviclash »

wawjgreen wrote: post #76

There's a little-known theory (whose name eludes me) that states: any outcome in behaviour is highly dependent on the immediate and unpredictable interplay of various environment variables and their real and perceived effect on the person. this alone cancels the efficacy of any nudges (but some variables may be in the same direction as the nudge--hence the original but misguided nudge research: they were fooled by ra…

Environment variables are mostly underplayed in social science, but why?

Re: No evidence for nudging after adjusting for publication bias

#205

Post by notafraudster »

lmeyerov wrote: post #48
notafraudster wrote: post #23

Earlier quoted context omitted.

The easiest to understand diagnostic used to measure publication bias is the funnel plot. Suppose the true effect of interest is theta = 0.2. Then the observed effects in studies should be centered at 0.2; some will be higher, some will be lower. Assuming no systematic error, the degree to which study results vary around 0.2 should be proportional to the precision of the study (think sampling error given a sampling d…

Interesting -- the problem may be a misapplication of funnel plots for metaanalysis. I'm not sure what theta is representing and only skimmed the paper, but especially in social scenarios and across social papers, seems unlikely to assume the same distributions and parameters across tasks & populations. Sometimes comes down to 'is there any effect??' and sometimes a precise notion of effect size in a lucky/clever spe…

Both the authors of this study and meta-analysts generally have followup responses and other diagnostics and models that work under degraded assumptions, so I don't think it makes a ton of sense to respond to "this is the absolute most introductory example of why this kind of intuition works" with "it doesn't work in case". There are a variety of ways to think through what data looks like under publication bias or not irrespective of what the "correct" effect would be.

I'd also add that this is a paper responding to an existing meta-analysis, so the claim that it's impossible to get a quantity of interest for a meta-analysis because the constituent studies are being inappropriately aggregated is itself an argument in favour of this paper's rejection of the original paper's finding.

I personally just rejected an m-a at a social science journal on thursday because I thought it suffered from unaddressed garbage-in garbage-out along the lines you mention (non-experimental data, no attempt to pin down causal identification, inadequate qualitative discussion of risks of publication bias, unclear QOI), so I am sympathetic to the criticisms, but just know that there are next steps. :)

Re: No evidence for nudging after adjusting for publication bias

#206

Post by raxxorraxor »

a_simm wrote: post #37

Wow. As a former social scientist with an axe to grind this hits hard. I like to provide that HN community with some context as to what this means. There are some 300 “research” departments in each of the major social sciences: psychology, sociology, economics and anthropology. If you believe what they say, about half of their mission is teaching and the other half is research. That’s a lot, tens of billions of dolla…

I think the worst offense of the social sciences recently is their quest to correct bias and with that the creation of idiots that believes themselves to be able to do that.

Because now you have said idiots running around screaming how terrible that bias is completely neglecting the fact that everyone is subjected to it.

Re: No evidence for nudging after adjusting for publication bias

#207

Post by richdougherty »

JusticeJuice wrote: post #202
richdougherty wrote: post #154

Earlier quoted context omitted.

I think all of those things you listed are kinds of nudges. They are changes in the "choice architecture" that steer the user to an action. In the context of government, a nudge means influencing people to choose something desirable, while still leaving open the option for people to choose what they want (hence preserving liberty). In contrast, a non-nudge solution would be a law or regulation that forces people into…

A nudge has to push the user towards one certain decision over another, that's the whole point. It's opinionated. The factors I listed aren't inherently opinionated, we could've tried to improve them without pushing the user to a specific choice: E.g. speed: We could've removed earlier parts of the onboarding to make the overall experience less long, or compacted the UI so it was visually easier to skim the choices.…

There are some good points you raise, and I think you're really testing the nuance of what nudging is. But nudges are perhaps more basic than you're thinking.

Nudges don't need to steer the user to a specific choice, just a behaviour change. Sticking with a conversion flow counts as a behaviour change.

Nudges don't need to be simple or understandable. They can be a set of complex changes where causation isn't clear. They just need to get results.

The only really hard requirement that would rule out a nudge is if you forced a choice or used financial incentives.

If you read the Nudge book you'll see that it's a political book, really. The authors introduce nudges as an alternative to hard regulation. Instead they propose that governments consider influencing behaviour in a softer way, but still leave the escape hatch open for people with strong preferences to choose what they want. This strikes a balance between state involvement and principles of liberty. (Or at least that's their argument.)

Because of this framing a nudge is defined mostly by what it isn't. It's not a nudge if it forces a user to a choice; a nudge is anything you do that changes what users do without forcing them.

This is what you've done with your series of changes that resulted in increased conversion. You've left all the choices open still, so users have as much freedom as before, but you've managed to predictably change user behaviour in a way that aligns with your goals. In other words, you've nudged them.

Re: No evidence for nudging after adjusting for publication bias

#208

Post by richdougherty »

Izkata wrote: post #185
richdougherty wrote: post #154

Earlier quoted context omitted.

I think all of those things you listed are kinds of nudges. They are changes in the "choice architecture" that steer the user to an action. In the context of government, a nudge means influencing people to choose something desirable, while still leaving open the option for people to choose what they want (hence preserving liberty). In contrast, a non-nudge solution would be a law or regulation that forces people into…

> that alters people's behavior I've always understood this part of the description to be more than a single one-off choice, so none of that around the decision point in the financial product would count.

I think any alteration to behaviour is open to nudging, but perhaps you can explain more?

Re: No evidence for nudging after adjusting for publication bias

#209

Post by raxxorraxor »

topaz0 wrote: post #151
rjmunro wrote: post #83

Reading this, I don't get how you can take all "nudging" and declare "No evidence". Surely "nudging" encompasses a whole range of different actual actions. Some nudges work, some don't. You can't just average across all of them. I'm probably totally misunderstanding, but it sounds similar to saying "there is no evidence for medicine" because you've averaged all the papers describing medical interventions that work an…

This is my interpretation as well. Also weird to think about publication bias in this way: "these studies about the effectiveness of snake oil as a drug weren't published, so we must be overestimating how effective drugs are". The authors do mention that there is likely to be heterogeneity in (real) effect sizes, but somehow still go with this title/abstract. Maybe there is a valid conclusion that some of the many nu…

> but somehow still go with this title/abstract.

While it might be to catch the reader, I don't think it is wrong. The large problem is that we had policies implemented to nudge people in certain directions. Apart from the ethical question there needs to be hard evidence before we employ authoritarianism like that. So the headline should be pardoned, but not those that employ nudging for the time being.

Re: No evidence for nudging after adjusting for publication bias

#210

Post by jjgreen »

inglor_cz wrote: post #102
xyzzy123 wrote: post #60

Earlier quoted context omitted.

If defaults don't work then Google wasted 15 billion dollars last year paying Apple to be in the search bar... I guess there must be further detail in the paper and I will have to read it to understand the nuance.

Defaults very clearly work in matters such as consent to organ donation. In countries where you need to opt out of organ donation, few people bother to do so. Another question is whether this increases the total amount of successful donations. I was looking around for studies and found this one [1], which basically says "in some countries, yes". [1] https://behavioralpolicy.org/wp-content/uploads/2020/01/Does...

Without consent, it's not a donation. "Harvesting" would be an appropriate name.
Post reply on HN