A/B testing original design vs. random template bought from ThemeForest
1–10 of 47 posts
Re: A/B testing original design vs. random template bought from ThemeForest
#2 During this time there were a total of 67 new
subscriptions. Of these 58% (39) came from the
new design and 42% (28) came from the old design.
Looks like the new one is a clear winner.
Using R's prop test, I get a p value of 0.22. (Type "prop.test(39,67)" to calculate it).
I think this means that in a world where it makes no difference which design is used, you would get a result as significant as this 22% of the time.
An alternative is the Adjusted Wald method. You can try it online here:
Which gives some confidence intervals which also range from "could be better" to "could be worse". Even when you reduce the confidence level from the typical 95% to 90%.
a quick check with an A/B testing calculator
even says that this result has significance
(~90% likely)
Re: A/B testing original design vs. random template bought from ThemeForest
#3During this time there were a total of 67 new subscriptions. Of these 58% (39) came from the new design and 42% (28) came from the old design. Looks like the new one is a clear winner. Is it? This seems a small population to settle on a clear winner. Using R's prop test, I get a p value of 0.22. (Type "prop.test(39,67)" to calculate it). I think this means that in a world where it makes no difference which design is…
Reminds me of: "A/A Testing: How I increased conversions 300% by doing absolutely nothing" " rel="nofollow">https://kadavy.net/blog/posts/aa-testing/>
Re: A/B testing original design vs. random template bought from ThemeForest
#4During this time there were a total of 67 new subscriptions. Of these 58% (39) came from the new design and 42% (28) came from the old design. Looks like the new one is a clear winner. Is it? This seems a small population to settle on a clear winner. Using R's prop test, I get a p value of 0.22. (Type "prop.test(39,67)" to calculate it). I think this means that in a world where it makes no difference which design is…
[deleted]
Re: A/B testing original design vs. random template bought from ThemeForest
#5During this time there were a total of 67 new subscriptions. Of these 58% (39) came from the new design and 42% (28) came from the old design. Looks like the new one is a clear winner. Is it? This seems a small population to settle on a clear winner. Using R's prop test, I get a p value of 0.22. (Type "prop.test(39,67)" to calculate it). I think this means that in a world where it makes no difference which design is…
Whatever test you do, you would need to know the total number of visitors in each group, right?
And unless I missed it the article doesn't state those numbers.
Intuitively, the numbers you quoted would be more significant the bigger the test and control groups are.
Re: A/B testing original design vs. random template bought from ThemeForest
#6During this time there were a total of 67 new subscriptions. Of these 58% (39) came from the new design and 42% (28) came from the old design. Looks like the new one is a clear winner. Is it? This seems a small population to settle on a clear winner. Using R's prop test, I get a p value of 0.22. (Type "prop.test(39,67)" to calculate it). I think this means that in a world where it makes no difference which design is…
I used http://www.abtestcalculator.com/ and entered 3000 participants -> 28 conversions and 3000 participants -> 39 conversions.
I neglected to record how many views each version had, but should be at least 3000 since the conversion ratio is about 0.5 - 1%.
Re: A/B testing original design vs. random template bought from ThemeForest
#7During this time there were a total of 67 new subscriptions. Of these 58% (39) came from the new design and 42% (28) came from the old design. Looks like the new one is a clear winner. Is it? This seems a small population to settle on a clear winner. Using R's prop test, I get a p value of 0.22. (Type "prop.test(39,67)" to calculate it). I think this means that in a world where it makes no difference which design is…
That's not how you use prop.test. What you've tested using that invocation is the null hypothesis that the underlying probability of 39/67 is 0.5.
If you want to perform a test of a difference of two proportions, you need to do:
prop.test(c(39, 67), c(total_group_a_impressions, total_group_b_impressions))
I don't have experience with A/B testing, so I'm not sure if this is typically or best handled using this particular statistical test.
Edit: The first parameter should be c(39, 28), meaning the total conversions in each group. I have no excuse beyond being tired.
Edit 2: To clarify, I think he should still use the two-sample form of prop.test, especially since we did not know at the time of his posting that the sample sizes are equal.
Re: A/B testing original design vs. random template bought from ThemeForest
#8During this time there were a total of 67 new subscriptions. Of these 58% (39) came from the new design and 42% (28) came from the old design. Looks like the new one is a clear winner. Is it? This seems a small population to settle on a clear winner. Using R's prop test, I get a p value of 0.22. (Type "prop.test(39,67)" to calculate it). I think this means that in a world where it makes no difference which design is…
I used http://www.abtestcalculator.com/ and entered 3000 participants -> 28 conversions and 3000 participants -> 39 conversions. I neglected to record how many views each version had, but should be at least 3000 since the conversion ratio is about 0.5 - 1%.
Resource provided uses a very naive approach to determining the outcome of an AB test. It's not accurate, given the very small numbers.
Re: A/B testing original design vs. random template bought from ThemeForest
#9Earlier quoted context omitted.
I used http://www.abtestcalculator.com/ and entered 3000 participants -> 28 conversions and 3000 participants -> 39 conversions. I neglected to record how many views each version had, but should be at least 3000 since the conversion ratio is about 0.5 - 1%.
Resource provided uses a very naive approach to determining the outcome of an AB test. It's not accurate, given the very small numbers.
Yes, I don't have much data to work with, and was also surprised that the calculator considered this significant. But even without significance, I assume it still makes sense to go with the winner?
Re: A/B testing original design vs. random template bought from ThemeForest
#10During this time there were a total of 67 new subscriptions. Of these 58% (39) came from the new design and 42% (28) came from the old design. Looks like the new one is a clear winner. Is it? This seems a small population to settle on a clear winner. Using R's prop test, I get a p value of 0.22. (Type "prop.test(39,67)" to calculate it). I think this means that in a world where it makes no difference which design is…
I used http://www.abtestcalculator.com/ and entered 3000 participants -> 28 conversions and 3000 participants -> 39 conversions. I neglected to record how many views each version had, but should be at least 3000 since the conversion ratio is about 0.5 - 1%.
The statement I see on that page is "There is a 91% chance that Variation A has a higher conversion rate".
I am not sure how to interpret that. We would have to dive into the GitHub repo and figure out which test it performs I guess.