Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Winning A/B results were not translating into improved user acquisition

blog.sumall.com

Post reply on HN
Print view View original

61–67 of 67 posts

Re: Winning A/B results were not translating into improved user acquisition

#61

Post by jules »

yummyfajitas wrote: post #60
jules wrote: post #58

Earlier quoted context omitted.

You can just show the posterior and let your brain be the decision rule. You can visually see the difference in conversion rate and the uncertainty around it. That info makes it easy to decide whether to continue the test or stop the test and pick the best performer. Much better information to base a decision on than a hypothesis test with a significance threshold that people pull out of their ass. If you want to be…

I didn't say hypothesis test , I said decision rule . The method I describe in the article has only two quantities "pulled out of the ass" - the threshold of caring and the prior. If you visually inspect the posterior, your are implicitly pulling out of your ass an unknown "threshold of visual similarity". That article is weird. It uses a normal distribution as the prior for the conversion rate. That's incorrect. Fro…

> I didn't say hypothesis test, I said decision rule.

I did not say that you said hypothesis test.

> If you visually inspect the posterior, your are implicitly pulling out of your ass an unknown "threshold of visual similarity".

Yes, but you are "implicitly pulling a number out of your ass" based on a lot more information. When you ask somebody to come up with a mechanical decision rule before seeing the posterior, it's unlikely that you will get as good a decision as when you just show them the posterior.

> That's incorrect. From the article: "To begin we will choose a Beta distribution prior." The computational intensiveness is not caused by the choice of prior, it's caused by the need to evaluate an integral over the joint posterior.

Ah, I was confused because they are specifying the prior in terms of a mean and standard deviation. That is a very weird way to represent a beta distribution.

> The computational intensiveness is not caused by the choice of prior, it's caused by the need to evaluate an integral over the joint posterior.

I see, they are computing expected_value(max(ctr[A]-ctr[B], 0.0)). That is still weird though. What you want to know is if it's worth it to run the test another time. So you want to compare E(final conversion rate if stop now) with E(final conversion rate if run another time), and if the latter is not much greater than the former you stop the test. Both of those have a closed form. Even better would be to compare E(final conversion rate if stop now) and E(final conversion rate if we test A) and E(final conversion rate if we test B). Then you would also automatically decide the best version to show (e.g. if the uncertainty about A is small and the uncertainty about B is big, you'll show B).

> A Dirichlet prior is also not what you'd use for more than 2 alternatives

Hm? Lets say you have a free plan, basic plan, and enterprise plan. This is a very common scenario in practice. A dirichlet prior would be the natural thing to use here, IMO.

Re: Winning A/B results were not translating into improved user acquisition

#62

Post by yummyfajitas »

jules wrote: post #61
yummyfajitas wrote: post #60

Earlier quoted context omitted.

I didn't say hypothesis test , I said decision rule . The method I describe in the article has only two quantities "pulled out of the ass" - the threshold of caring and the prior. If you visually inspect the posterior, your are implicitly pulling out of your ass an unknown "threshold of visual similarity". That article is weird. It uses a normal distribution as the prior for the conversion rate. That's incorrect. Fro…

> I didn't say hypothesis test, I said decision rule. I did not say that you said hypothesis test. > If you visually inspect the posterior, your are implicitly pulling out of your ass an unknown "threshold of visual similarity". Yes, but you are "implicitly pulling a number out of your ass" based on a lot more information. When you ask somebody to come up with a mechanical decision rule before seeing the posterior, i…

E(final conversion rate if run another time)... Both of those have a closed form.

I'm curious - where can I learn more?

Lets say you have a free plan, basic plan, and enterprise plan...A dirichlet prior would be the natural thing to use here, IMO.

This would be handled via Dirichlet, and then the results multiplied by their LTV. I thought you were referring to multiple variants - i.e., landing page A, landing page B, landing page C.

Re: Winning A/B results were not translating into improved user acquisition

#63

Post by jules »

yummyfajitas wrote: post #62
jules wrote: post #61

Earlier quoted context omitted.

> I didn't say hypothesis test, I said decision rule. I did not say that you said hypothesis test. > If you visually inspect the posterior, your are implicitly pulling out of your ass an unknown "threshold of visual similarity". Yes, but you are "implicitly pulling a number out of your ass" based on a lot more information. When you ask somebody to come up with a mechanical decision rule before seeing the posterior, i…

E(final conversion rate if run another time)... Both of those have a closed form. I'm curious - where can I learn more? Lets say you have a free plan, basic plan, and enterprise plan...A dirichlet prior would be the natural thing to use here, IMO. This would be handled via Dirichlet, and then the results multiplied by their LTV. I thought you were referring to multiple variants - i.e., landing page A, landing page B,…

Suppose you have Beta(a1,b2) and Beta(a2,b2) at the current step. The expected conversion rates are:

    M(a,b) = a/(a+b)

    E1 = M(a1,b1)
    E2 = M(a2,b2)
If we stop now the expected conversion rate is E = max(E1,E2).

If we continue for another timestep with option 1 then the question is whether that can make us switch from 1 to 2 or from 2 to 1 or not. If it can't then the expected conversion rate is the same whether or not we execute one more step. Lets assume without loss of generality that option 2 is currently winning, but if option 1 gets another conversion then 1 is winning. So the new expected conversion rate is:

    E' = int(p_1(r)*(r*r + (1-r)*E2)), r=0..1)
where p_1 is the probability density of Beta(a1,b1). All the moments of the beta distribution have a closed form, so E' also has a closed form.

You could generalize this to running it for n more times instead of one more time, you'd get an expression of the form:

   E = int(p_1(r1)*p_2(r2)*polynomial(r1,r2))
I suspect that also has a closed form but I'm not sure at first glance.

Re: Winning A/B results were not translating into improved user acquisition

#64

Post by kareemm »

dshacker wrote: post #49
kareemm wrote: post #46

This title used to read "How Optimizely (Almost) Got Me Fired", which is the actual title of the article. It seems a mod (?) changed it to "Winning A/B results were not translating into improved user acquisition". I've seen a descriptive title left by the submitter change back to the less descriptive original by a mod. But I'm curious why a mod would editorialize certain titles and change them away from their origina…

I feel that the second title is better, as it talks about the kind of testing they are using, instead of being a click bait of "HOW DID IT GET YOU FIRED?".

My question is why mods change some headlines away from the originals to be more descriptive (good) and why they change back to the originals even though they are less descriptive (bad).

FWIW the change to this headline seems like the right decision to me.

Re: Winning A/B results were not translating into improved user acquisition

#65

Post by [deleted] »

kareemm wrote: post #46

This title used to read "How Optimizely (Almost) Got Me Fired", which is the actual title of the article. It seems a mod (?) changed it to "Winning A/B results were not translating into improved user acquisition". I've seen a descriptive title left by the submitter change back to the less descriptive original by a mod. But I'm curious why a mod would editorialize certain titles and change them away from their origina…

[deleted]

Re: Winning A/B results were not translating into improved user acquisition

#66

Post by dang »

kareemm wrote: post #64
dshacker wrote: post #49

Earlier quoted context omitted.

I feel that the second title is better, as it talks about the kind of testing they are using, instead of being a click bait of "HOW DID IT GET YOU FIRED?".

My question is why mods change some headlines away from the originals to be more descriptive (good) and why they change back to the originals even though they are less descriptive (bad). FWIW the change to this headline seems like the right decision to me.

The guideline is to use the original title unless it is misleading or linkbait [1]. It's astonishing how often that qualifier gets dropped from these discussions. It's pretty critical, and makes the reason for most title changes pretty obvious.

1. https://news.ycombinator.com/newsguidelines.html

Re: Winning A/B results were not translating into improved user acquisition

#67

Post by kareemm »

dang wrote: post #66
kareemm wrote: post #64

Earlier quoted context omitted.

My question is why mods change some headlines away from the originals to be more descriptive (good) and why they change back to the originals even though they are less descriptive (bad). FWIW the change to this headline seems like the right decision to me.

The guideline is to use the original title unless it is misleading or linkbait [1]. It's astonishing how often that qualifier gets dropped from these discussions. It's pretty critical, and makes the reason for most title changes pretty obvious. 1. https://news.ycombinator.com/newsguidelines.html

Thanks for the response. I'd humbly submit that there are occasions where the guidelines should be ignored in service of a more descriptive (non-linkbaity) title.

I can't find the submission but one recent example that comes to mind is a presentation on radar detectors that was fascinating. I clicked because the submitter described the article; the original title was (IIRC) the model number of the radar gun.

Later a mod changed the HN post back to the model number, which had zero relevance to anybody not in the radar gun industry.

Post reply on HN