Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

A/B test improved your website's conversion rate? Not so fast

blog.alexandervolkmann.com

Post reply on HN
Print view View original

61–68 of 68 posts

Re: A/B test improved your website's conversion rate? Not so fast

#61

Post by Maxion »

pottertheotter wrote: post #46
jspash wrote: post #45

Earlier quoted context omitted.

We stopped doing A/B tests after I insisted that they all be done as A/A/B tests. Suddenly the "clear winners" weren't so clear after all. It confused and frustrated the marketing department so much that it was decided just to stop doing them all together. The reason I wanted this type of test was because it was a waste of time testing shades of blue or two headlines that only differed by 2 words. The test variants w…

For an A/A/B test are you taking three samples (instead of two), and two of the three get shown the same thing (A)? Then you only consider the results for the B group if the two A groups show the same behavior?

You take three samples and two of three get shown the same thing. What happens here is both A groups will show different results until your sample grows enough that the CI window becomes small.

This helps to show the effect of a low sample from a non-uniform distribution.

A lot of people (me included) think they know statistics, but they don't.

The blogpost in OP also tries to explain the same thing - you shouldn't do statistics without understanding what it is your doing.

Re: A/B test improved your website's conversion rate? Not so fast

#62

Post by chunkyks »

aeternum wrote: post #60
chunkyks wrote: post #57

Earlier quoted context omitted.

You haven't convinced me that preventing browser tracking is making the internet "slightly worse overall". If sites are having trouble converting me, perhaps it's not me that's the problem.

The issue is most sites can no longer tell if they are converting you

It's not obvious to me that that is a problem for me, or that it makes the internet worse

Re: A/B test improved your website's conversion rate? Not so fast

#63

Post by lifeplusplus »

jefftk wrote: post #2

Summary: if you run an experiment where you try to rush users to convert, and you only run the experiment for a short time, it will look great even though it might be lossy overall, because you're capturing a larger proportion of conversions in the experiment group. You can also run into this sort of problem with user learning effects, where initially a large change in the UI can give a large change in behavior due t…

I don't get this summary how are you capturing a larger part of conversations

Re: A/B test improved your website's conversion rate? Not so fast

#64

Post by jefftk »

lifeplusplus wrote: post #63
jefftk wrote: post #2

Summary: if you run an experiment where you try to rush users to convert, and you only run the experiment for a short time, it will look great even though it might be lossy overall, because you're capturing a larger proportion of conversions in the experiment group. You can also run into this sort of problem with user learning effects, where initially a large change in the UI can give a large change in behavior due t…

I don't get this summary how are you capturing a larger part of conversations

Conversions, not conversations, if that helps?

Re: A/B test improved your website's conversion rate? Not so fast

#65

Post by bananamansion »

Maxion wrote: post #61
pottertheotter wrote: post #46

Earlier quoted context omitted.

For an A/A/B test are you taking three samples (instead of two), and two of the three get shown the same thing (A)? Then you only consider the results for the B group if the two A groups show the same behavior?

You take three samples and two of three get shown the same thing. What happens here is both A groups will show different results until your sample grows enough that the CI window becomes small. This helps to show the effect of a low sample from a non-uniform distribution. A lot of people (me included) think they know statistics, but they don't. The blogpost in OP also tries to explain the same thing - you shouldn't d…

dont you then need to run the experiment for a long long time to get the significance of it? + the site needs enough users viewing the page to run it in any reasonable amount of time.. i would think most sites dont get enough traffic to run a/a let alone ab?

Re: A/B test improved your website's conversion rate? Not so fast

#66

Post by antux »

rwilson4 wrote: post #22
antux wrote: post #18

A/B testing can be useful at times but it's largely overrated because you're only discovering the best design out of the ones you test. That means there could be a far better design that you failed include in the experiment. Just because one design converts more than the other doesn't mean it's the design with optimal UX. I've seen many tests where the designs included already had faulty UX. This is why it's better t…

I agree, but that's not the whole story. A/B testing isn't just for identifying the best option, it's for figuring out how much opportunity there is in a particular facet of your business. If you try a few reasonable designs, and some designs have much better performance than others, then it makes sense to continue investing time to further improve. You can take what you've learned and try to come up with even better…

That's my point with having a trained UX designer who knows what they're doing. They can improve a design much better than the average developer or designer because they have more training. An average designer or developer is going to hit a ceiling and be spinning their wheels creating variants with minimal improvement. This is why a/b testing is overrated. In other words, too much emphasis placed on testing, testing, testing and not enough on UX design.

Re: A/B test improved your website's conversion rate? Not so fast

#67

Post by volkale »

clircle wrote: post #53

It seems statistically dubious to me that the point k = infinity is in the sample space and that the model is well defined. Does this model require a proper prior on p? Cool blog post.

- Using the point k=infinity is how I would define it mathematically. In the Stan code I mapped that point to k=-1.

- I don't think a proper prior is required.

- Thank you :)

(Disclaimer: I am the author of the blog post.)

Re: A/B test improved your website's conversion rate? Not so fast

#68

Post by clircle »

volkale wrote: post #67
clircle wrote: post #53

It seems statistically dubious to me that the point k = infinity is in the sample space and that the model is well defined. Does this model require a proper prior on p? Cool blog post.

- Using the point k=infinity is how I would define it mathematically. In the Stan code I mapped that point to k=-1. - I don't think a proper prior is required. - Thank you :) (Disclaimer: I am the author of the blog post.)

On more reflection... the support of p is finite, so the prior ought to be proper.
Post reply on HN