Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

A/B test improved your website's conversion rate? Not so fast

blog.alexandervolkmann.com

Post reply on HN
Print view View original

51–60 of 68 posts

Re: A/B test improved your website's conversion rate? Not so fast

#51

Post by robomartin »

Well, this is where it comes down to understanding statistics. Yes, that subject we all hated in college and could not wait to pass and forget about. I think I can say that most A/B testing is, to be kind, statistically flawed. At the same time, it is possible to think that only some of the largest websites might have enough traffic to do it right (whatever that means).

And then there's the big question: How much of business did you lose in the process of arriving at what seems like an optimal solution (which might just be a local peak, rather than a global optimum point)?

That said, what's the alternative? To optimize, or not, that is the question.

Re: A/B test improved your website's conversion rate? Not so fast

#52

Post by dmkii »

omegalulw wrote: post #44
jefftk wrote: post #2

Summary: if you run an experiment where you try to rush users to convert, and you only run the experiment for a short time, it will look great even though it might be lossy overall, because you're capturing a larger proportion of conversions in the experiment group. You can also run into this sort of problem with user learning effects, where initially a large change in the UI can give a large change in behavior due t…

Wouldn't it be better to run the experiment longer _and_ discard the data from initial few weeks?

The current state of browser tracking preventions also means that you’re unlikely to identify conversions from the same user that saw your experiment after a week or sometimes even 24 hours.

Re: A/B test improved your website's conversion rate? Not so fast

#54

Post by aeternum »

dmkii wrote: post #52
omegalulw wrote: post #44

Earlier quoted context omitted.

Wouldn't it be better to run the experiment longer _and_ discard the data from initial few weeks?

The current state of browser tracking preventions also means that you’re unlikely to identify conversions from the same user that saw your experiment after a week or sometimes even 24 hours.

Yes, browser tracking prevention is one of those things that seems like a good idea at first but likely makes the internet slightly worse overall.

Sites can only optimize for what they can see and we've made it so they can only see short-term engagement.

Another is all the annoying cookie popups as a result of GDPR.

Re: A/B test improved your website's conversion rate? Not so fast

#55

Post by throw1234651234 »

ssharp wrote: post #43
temple__ wrote: post #40

Earlier quoted context omitted.

https://support.google.com/optimize/answer/7100284?hl=en

Anti-flicker snippet is definitely the first step. Since this is a SPA and the flicker may be caused well past the point of the initial pageview, there may also be an issue with how the code is written and hooked into the SPA framework, in this case React.

Thanks, we already tried it. Causes page load to go from .02 s to 3-4.

Re: A/B test improved your website's conversion rate? Not so fast

#57

Post by chunkyks »

aeternum wrote: post #54
dmkii wrote: post #52

Earlier quoted context omitted.

The current state of browser tracking preventions also means that you’re unlikely to identify conversions from the same user that saw your experiment after a week or sometimes even 24 hours.

Yes, browser tracking prevention is one of those things that seems like a good idea at first but likely makes the internet slightly worse overall. Sites can only optimize for what they can see and we've made it so they can only see short-term engagement. Another is all the annoying cookie popups as a result of GDPR.

You haven't convinced me that preventing browser tracking is making the internet "slightly worse overall".

If sites are having trouble converting me, perhaps it's not me that's the problem.

Re: A/B test improved your website's conversion rate? Not so fast

#58

Post by def_true_false »

aeternum wrote: post #54
dmkii wrote: post #52

Earlier quoted context omitted.

The current state of browser tracking preventions also means that you’re unlikely to identify conversions from the same user that saw your experiment after a week or sometimes even 24 hours.

Yes, browser tracking prevention is one of those things that seems like a good idea at first but likely makes the internet slightly worse overall. Sites can only optimize for what they can see and we've made it so they can only see short-term engagement. Another is all the annoying cookie popups as a result of GDPR.

The popups are a result of tracking, not GDPR. Websites without tracking don't need to have them.

It's somewhat amusing that the overlap of garbage content farms and sites with annoying consent popups is almost perfect. I wonder if it could be used for search engine ranking.

Re: A/B test improved your website's conversion rate? Not so fast

#59

Post by ComputerGuru »

pottertheotter wrote: post #46
jspash wrote: post #45

Earlier quoted context omitted.

We stopped doing A/B tests after I insisted that they all be done as A/A/B tests. Suddenly the "clear winners" weren't so clear after all. It confused and frustrated the marketing department so much that it was decided just to stop doing them all together. The reason I wanted this type of test was because it was a waste of time testing shades of blue or two headlines that only differed by 2 words. The test variants w…

For an A/A/B test are you taking three samples (instead of two), and two of the three get shown the same thing (A)? Then you only consider the results for the B group if the two A groups show the same behavior?

Not the person you're responding to, but yes, that's the idea. It's a control not for the the B but for the unknown unknowns that may or may not be there.

If A' and B both statistically differ from A, then you have a problem because you're not testing what you think you are testing, regardless of what your naive A/B test's p-value would have indicated.

Re: A/B test improved your website's conversion rate? Not so fast

#60

Post by aeternum »

chunkyks wrote: post #57
aeternum wrote: post #54

Earlier quoted context omitted.

Yes, browser tracking prevention is one of those things that seems like a good idea at first but likely makes the internet slightly worse overall. Sites can only optimize for what they can see and we've made it so they can only see short-term engagement. Another is all the annoying cookie popups as a result of GDPR.

You haven't convinced me that preventing browser tracking is making the internet "slightly worse overall". If sites are having trouble converting me, perhaps it's not me that's the problem.

The issue is most sites can no longer tell if they are converting you
Post reply on HN