Leaked UI A/B Tests from Major Websites
goodui.org
Leaked UI A/B Tests from Major Websites
1–10 of 17 posts
Re: Leaked UI A/B Tests from Major Websites
#2Would be great for a quiz to guess the result!
Re: Leaked UI A/B Tests from Major Websites
#3In many of these cases, the "A/B test" may have been accidental.
Running a software rollout is frequently done slowly, datacenter by datacenter, and during that time some people might see one version and others might see another.
From the users perspective it looks the same as an A/B test, but the difference is nobody was looking at the results...
Re: Leaked UI A/B Tests from Major Websites
#4I don’t think that in every case it is necessary down to user rejections. I can’t believe that in the netflix case users actually prefer to have to login with two post backs, one for the login, one for the password.
Re: Leaked UI A/B Tests from Major Websites
#5I don’t think that in every case it is necessary down to user rejections. I can’t believe that in the netflix case users actually prefer to have to login with two post backs, one for the login, one for the password.
This is a necessary pattern for increased security.
Re: Leaked UI A/B Tests from Major Websites
#6I don’t think that in every case it is necessary down to user rejections. I can’t believe that in the netflix case users actually prefer to have to login with two post backs, one for the login, one for the password.
This appears to be the registration flow, which makes more sense.
Re: Leaked UI A/B Tests from Major Websites
#7Where is the data on results? I looked at an AirBnB “experiment”, they moved an action button above the fold (duh). But no details on how much more effective the move was.
I am all for A/B testing, but the devil is in the details. You can get more users tapping the purchase by moving the purchase button where users are more prone to accidentally tap the purchase button. That doesn’t mean you get more purchases, or that the move was a positive change.
Re: Leaked UI A/B Tests from Major Websites
#8I don’t think that in every case it is necessary down to user rejections. I can’t believe that in the netflix case users actually prefer to have to login with two post backs, one for the login, one for the password.
”I can’t believe...” - this is why people run A/B tests.
Re: Leaked UI A/B Tests from Major Websites
#9Where is the data on results? I looked at an AirBnB “experiment”, they moved an action button above the fold (duh). But no details on how much more effective the move was. I am all for A/B testing, but the devil is in the details. You can get more users tapping the purchase by moving the purchase button where users are more prone to accidentally tap the purchase button. That doesn’t mean you get more purchases, or th…
I don't think they have results. It looks like they are regularly scraping sites and looking for diffs across users. So you can say what the test was, how long it ran for roughly, and whether they kept or rejected it, but you have no way of knowing what the quantitative results are (aside from whatever inferences you can make from estimating the tested _n_ and then assuming they are using an efficient testing procedure with optional stopping / bandits plus the final choice to infer upper/lower bounds on the effect size).
Re: Leaked UI A/B Tests from Major Websites
#10I don’t think that in every case it is necessary down to user rejections. I can’t believe that in the netflix case users actually prefer to have to login with two post backs, one for the login, one for the password.
That's actually not a login but a signup form. More people will probably enter anything at all with only one field showing. And since they've already interacted with the site, they might feel inclined to complete the process even after they find out it's multi-step. Some will of course bounce because of it, but it should be net positive overall.