Simulating A/B tests offline with counterfactual inference
1–5 of 5 posts
Re: Simulating A/B tests offline with counterfactual inference
#2> To mitigate presentation bias, we can always pick a small fraction of users that are always shown uniformly random results
Would it be equivalent to displaying a small number of random (exploration) items to each user in addition to algo recommended (exploitation) items, which seems more palatable?
Re: Simulating A/B tests offline with counterfactual inference
#3As long as you know which items are random, and you can compute the probability of each item (regardless of whether it's uniform random or through your algorithm) for each user, the method still applies.
While it does seem more palatable to just show a maximum of X% uniformly random items to all users, the computation of p(y|x) can get a bit hairy for any item y, because now it is a product of a uniform distribution and your stochastic ranking algorithm. If you _can_ compute that, you'd be fine.
Re: Simulating A/B tests offline with counterfactual inference
#4> To mitigate presentation bias, we can always pick a small fraction of users that are always shown uniformly random results Would it be equivalent to displaying a small number of random (exploration) items to each user in addition to algo recommended (exploitation) items, which seems more palatable?
Thanks for your question!
As long as you can compute the probability of each item being shown (regardless of whether it's uniform random or through your algorithm) to each user, the method still applies. While it does seem more palatable to just show a maximum of X% uniformly random items to all users, the computation of p(y|x) can get a bit hairy for any item y, because now it is a product of a uniform distribution and your stochastic ranking algorithm. If you _can_ compute that, you'd be fine.
Re: Simulating A/B tests offline with counterfactual inference
#5Awesome read!