Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

No evidence for nudging after adjusting for publication bias

pnas.org

Post reply on HN
Print view View original

191–200 of 246 posts

Re: No evidence for nudging after adjusting for publication bias

#191

Post by rad88 »

concordDance wrote: post #160
rad88 wrote: post #137

Earlier quoted context omitted.

The hacker news banner color doesn't matter and few have ever wanted to change it. But your financial position and needs, what % of your salary you can afford to money-hole until retirement, does matter and is pretty individual. It doesn't defy credulity to me that generally people would make a choice about this (when can I retire?), and that the default doesn't influence it. I grant that it would be surprising if it…

The default influenced me and pretty much everyone in my company Ive talked to (namely, almost everyone stuck to it). So yes, it defies credulity.

I believe you. If you know it influenced you though, that means you didn't ignore it or not even realize you could change it, which is the idea I was replying to.

Again, it would be surprising if it didn't matter at all, but not unimaginable. What you're saying is that almost everybody in your company would have contributed a lesser amount if not for the default. It means you can all afford to give up $20k or whatever in income this year. There are other factors.

There is no truth to the matter of "whether defaults change behavior". This thread started about 401ks and then was taken into color preferences on the web. If someone has a gun to their head is asked if they want to die, I'm sure we'll agree that whatever the default is doesn't matter. Whether defaults do anything depends on what we're talking about. Nudges might work in web ux but not economics, why is that so incredulous?

Re: No evidence for nudging after adjusting for publication bias

#192

Post by saalweachter »

asdff wrote: post #136
rsanek wrote: post #123

Earlier quoted context omitted.

I mean, yeah, if she has solid RCT data on thousands to millions of childbirths and has found a statistically significant impact from using the magic crystals, I would support their use. A/B as well as scientific research uses the same basis. The issue is that in fact the midwife will not have such data. The comparison being made is that A/B testing, if run competently, is pretty close to scientific research, in part…

I wonder how many engineers crack open a statistics book to find the correct test versus just plotting box plots and saying "see looks pretty different"

To be fair, the more profound a result the less math you need to convince anyone it is the case.

Re: No evidence for nudging after adjusting for publication bias

#193

Post by unityByFreedom »

rayiner wrote: post #188
unityByFreedom wrote: post #187

Earlier quoted context omitted.

> They don't preregister their trials. First of all, preregistration is not a requirement for the scientific method, which has functioned well for centuries. That is a recent trend in response to the overflowing amount of haphazardly published science. Second, it is up to the individual scientist to decide to preregister or not. Some social scientists may preregister. Third, small sample size may be a fair critique,…

> You've made a lot of generalizations here that amount to, "social scientists aren't as rigorous as other areas of science, therefore we should only believe studies that disagree with their results". I think it’s more “ignore them completely.” It brings “science” into disrepute to let social science associate with the other sciences. > I don't think throwing the baby out with the bath water is helpful. There is no b…

> It brings “science” into disrepute to let social science associate with the other sciences.

So you just want it renamed to "social studies" or what? What is your proposal, that nobody research this topic, or that they be separated in journals etc? I doubt that will have much impact on whether it makes the news. If you want that to change, you may need to get yourself onto the board of a journal you care about.

> There is no baby!

That's reductive. Just because you don't see the baby doesn't mean it doesn't exist.

> Doesn’t that prove the point? It wasn’t science then and isn’t science today.

No, it just proves it's an old disagreement, like nature vs nurture.

There is plenty of work in social science that contributes to humanity. It will always have smaller sample sizes due to the nature of collecting the data. The work can be considered useful nonetheless.

Re: No evidence for nudging after adjusting for publication bias

#195

Post by ad8e »

pizzathyme wrote: post #167
TameAntelope wrote: post #148

Earlier quoted context omitted.

Isn't this just one specific analysis using a very narrow definition of "nudge", one that doesn't even begin to encompass the work being done at those "300 "research" departments"? Further, isn't this using the same data from the original meta analysis that did find "of small to medium size" effect[0]? Why would this, alone, undo decades of research and clear, bright-line conclusions such as the ones cited in my sibl…

I agree, I have worked at 5 separate tech companies and have conducted hundreds of statistically significant experiments, changing what people select by default. These methods have been effective at helping hundreds of millions of people improve choices. It’s common practice, think: pricing on Amazon, default tips on Uber, default purchase price in a video game, etc. I don’t see how this would make me reinterpret all…

One interpretation is that it is very hard to extract value from the nudge literature. When reading research articles, one must estimate and adjust for biases. This adjustment shifts the p-values by unclear amounts. So a positive result may just be a fluke.

As for your own AB tests, you have seen the processes that go into them and do not need to adjust for unknown biases. So when they demonstrate a nudge effect, you can believe it.

Re: No evidence for nudging after adjusting for publication bias

#196

Post by efitz »

pizzathyme wrote: post #167
TameAntelope wrote: post #148

Earlier quoted context omitted.

Isn't this just one specific analysis using a very narrow definition of "nudge", one that doesn't even begin to encompass the work being done at those "300 "research" departments"? Further, isn't this using the same data from the original meta analysis that did find "of small to medium size" effect[0]? Why would this, alone, undo decades of research and clear, bright-line conclusions such as the ones cited in my sibl…

I agree, I have worked at 5 separate tech companies and have conducted hundreds of statistically significant experiments, changing what people select by default. These methods have been effective at helping hundreds of millions of people improve choices. It’s common practice, think: pricing on Amazon, default tips on Uber, default purchase price in a video game, etc. I don’t see how this would make me reinterpret all…

How much of this is “nudging” vs. “clearly explaining trade-offs”?

I’ve not done any rigorous research, but I’ve participated in projects that resulted in dramatic shifts towards customers choosing what the dev team thought was the “best” outcome, just by altering wording, or making “dangerous” choices harder (such as by requiring more clicks to enable).

Re: No evidence for nudging after adjusting for publication bias

#197

Post by hackerlight »

IshKebab wrote: post #77
hackerlight wrote: post #5

Earlier quoted context omitted.

You are misunderstanding. They're saying that there's an absence of evidence, not evidence of absence.

Absence of evidence is evidence of absence. It's just not proof of absence. For example, if I search your entire house for drugs, using drug sniffing dogs and so on and I don't find any at all, that's pretty good evidence that you don't have any. It's not proof though - you might have just stashed them really well. Similarly, if people have been looking for nudge effects for ages, doing loads of studies on it for yea…

Well said. I'd just change your first sentence to "Absence of evidence can be (but isn't necessarily) evidence of absence.", which is more in line with the rest of your post.

Re: No evidence for nudging after adjusting for publication bias

#198

Post by JusticeJuice »

civilized wrote: post #115
JusticeJuice wrote: post #113

Earlier quoted context omitted.

I was working on a fintech project (gonna be vague as it's not yet released). The legal team told us we couldn't use default choices anywhere, as it could count as giving financial advice. Fair enough. So we designed the onboarding, and there was this choice the user had to make before we could create their account. During testing, we found people were getting really stuck on this choice, to the point of giving up. T…

Very interesting. One question: > The new design was faster. There was less screens with simpler choices. It went from a 'pick one of 5' to a 'heres the default, would you like to change it?'. Was it just the speed that made a difference? If you're just going from "pick one of 5" to "pick one of 5 but there's a default", I wouldn't expect one or the other to be "faster". Was the new design more different than that? A…

The first version was a screen with 5 choices, and detail about each choice that you'd have to scroll through. The second version was a simple "We've set this up for you" screen with two options, continue or customise. If you hit customise, you'd get shown the original five choice screen.

What I mean by it being faster is you could get to the next step of the process with both reading less text, and seeing less choices (just two buttons not 5). Cause if you just slapped the continue button (which most people did), you'd skip the whole explanation of all the choices.

Re: No evidence for nudging after adjusting for publication bias

#199

Post by istjohn »

unityByFreedom wrote: post #193
rayiner wrote: post #188

Earlier quoted context omitted.

> You've made a lot of generalizations here that amount to, "social scientists aren't as rigorous as other areas of science, therefore we should only believe studies that disagree with their results". I think it’s more “ignore them completely.” It brings “science” into disrepute to let social science associate with the other sciences. > I don't think throwing the baby out with the bath water is helpful. There is no b…

> It brings “science” into disrepute to let social science associate with the other sciences. So you just want it renamed to "social studies" or what? What is your proposal, that nobody research this topic, or that they be separated in journals etc? I doubt that will have much impact on whether it makes the news. If you want that to change, you may need to get yourself onto the board of a journal you care about. > Th…

It's not useful. In fact, it's actively harmful. The constant churn of findings and retractions undermines the public's belief in science. When people stop believing in science, you get things like flat earthers, global warming deniers, and vaccine skeptics.

Re: No evidence for nudging after adjusting for publication bias

#200

Post by unityByFreedom »

istjohn wrote: post #199
unityByFreedom wrote: post #193

Earlier quoted context omitted.

> It brings “science” into disrepute to let social science associate with the other sciences. So you just want it renamed to "social studies" or what? What is your proposal, that nobody research this topic, or that they be separated in journals etc? I doubt that will have much impact on whether it makes the news. If you want that to change, you may need to get yourself onto the board of a journal you care about. > Th…

It's not useful. In fact, it's actively harmful. The constant churn of findings and retractions undermines the public's belief in science. When people stop believing in science, you get things like flat earthers, global warming deniers, and vaccine skeptics.

The quibble here is whether social science is a science. Here's a good overview, including a video of Feynman which the HN crowd may appreciate:

https://www.quora.com/Is-social-science-a-real-science/answe...

Humanity is observable. It's just hard to collect the data. I think you can make the case that some science isn't as rigorous, or that the jury is still out, but to say that it isn't science at all is wrong in my opinion. Even Feynman acknowledges that conclusions may be drawn later.

Aristotle philosophized quite a bit and is considered an early contributor to the scientific method. More food for thought:

https://blogs.scientificamerican.com/cross-check/is-social-s...

Post reply on HN