Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

The persistent mischaracterization of Google and Facebook A/B tests

sciencedirect.com

Post reply on HN
Print view View original

11–16 of 16 posts

Re: The persistent mischaracterization of Google and Facebook A/B tests

#11

Post by robertlagrant »

gdc3000 wrote: post #3

Really found this paper interesting and concerning. I don’t work in marketing or run these kinds of studies. I do work at Qualtrics and have experience with A/B testing, in general. For those who work in this space and can relate to this paper, would it be helpful if Qualtrics developed some kind of audit panel in our product to help surface potential platform bias? For example, sample ratio mismatch or metadata bala…

Perhaps something that highlights the limited scope of a test's predictive power? E.g. for a test run on Facebook "This test is likely to be very useful for a another Facebook ad campaign with the same parameters, and at least somewhat useful for a Google ad campaign with equivalent parameters".

Re: The persistent mischaracterization of Google and Facebook A/B tests

#12

Post by aaron695 »

Ozzie_osman wrote: post #5

So their argument is that the tests are not providing causal inference, because the platforms can target the two A/B test groups differently. With that in mind, my reading of this is: if you're a researcher trying to say "Advertisement A is more appealing than Y", you need causal inference and these tests won't give them to you. If you're a marketer just trying to determine what ad spend is more efficient, you don't…

[dead]

Re: The persistent mischaracterization of Google and Facebook A/B tests

#13

Post by wbl »

charcircuit wrote: post #7
wbl wrote: post #6

Earlier quoted context omitted.

The reason you need this is that the hypothesis is about the creative not the combination with the platform. You'd want the creative choice to go the same way across platforms.

>You'd want the creative choice to go the same way across platforms. While it would be convenient, platforms aren't the same. You can't just assume people will react the same across platforms.

Right that's a different critique of the whole methodology that also should get made.

Re: The persistent mischaracterization of Google and Facebook A/B tests

#14

Post by Ozzie_osman »

tsimionescu wrote: post #8
Ozzie_osman wrote: post #5

So their argument is that the tests are not providing causal inference, because the platforms can target the two A/B test groups differently. With that in mind, my reading of this is: if you're a researcher trying to say "Advertisement A is more appealing than Y", you need causal inference and these tests won't give them to you. If you're a marketer just trying to determine what ad spend is more efficient, you don't…

I think it's somewhat more important than that. Basically the point is that the A/B tests tell you "which ad is more effective spending on this particular platform with the particular arrangement of users you selected". If you try to expand the user group after the A/B test is over, or if you take the ads to another platform, you shouldn't expect the results to hold. So if you're an ad exec, you should make sure you…

True, but any ad exec worth their salt already knows this, if not because of different targeting algorithms, then at least different user and intent profiles (eg social users are generally younger and lower intent).

Re: The persistent mischaracterization of Google and Facebook A/B tests

#15

Post by ano-ther »

Ozzie_osman wrote: post #5

So their argument is that the tests are not providing causal inference, because the platforms can target the two A/B test groups differently. With that in mind, my reading of this is: if you're a researcher trying to say "Advertisement A is more appealing than Y", you need causal inference and these tests won't give them to you. If you're a marketer just trying to determine what ad spend is more efficient, you don't…

The issue seems to be that the platforms optimize before showing — presumably because they get paid for click-throughs.

Couldn’t they offer an unbiased randomization option with a different payment model (eg based on showings, not clicks)? Would preserve their revenue and researchers get a good tool.

Re: The persistent mischaracterization of Google and Facebook A/B tests

#16

Post by tsimionescu »

Ozzie_osman wrote: post #14
tsimionescu wrote: post #8

Earlier quoted context omitted.

I think it's somewhat more important than that. Basically the point is that the A/B tests tell you "which ad is more effective spending on this particular platform with the particular arrangement of users you selected". If you try to expand the user group after the A/B test is over, or if you take the ads to another platform, you shouldn't expect the results to hold. So if you're an ad exec, you should make sure you…

True, but any ad exec worth their salt already knows this, if not because of different targeting algorithms, then at least different user and intent profiles (eg social users are generally younger and lower intent).

True, but any researcher worth their salt should also have already known this. And yet, lots of scientists didn't, and I'd bet the same is true for lots of ad execs.
Post reply on HN