> Statistics are hard for most people to understand. True, but that's exactly what statistics helps with, though also hard to understand. :)
People are hard for statistics to understand.
Lines of code that beat A/B testing (2012)
121–130 of 180 posts
Re: Lines of code that beat A/B testing (2012)
#122That's a statistically valid approach. Technically correct, the best kind of correct. Meanwhile, if your users get presented a different button whenever they come by, because the MAB is still pursuing its hill climbing, they'll rightfully accuse you of having extremely crappy UX. (And, sure, you can have MAB with user stickiness, but now you do need to talk about sampling bias) And MAB hill climb doesn't work at all…
A/B testing has the same problem unless you figure out how to treat the same user the same way each time. But yeah I generally don't give much credence to an assertion like this that isn't based on a real-world experience. It's not even like "this algo might help," it's "will beat A/B testing every time."
Re: Lines of code that beat A/B testing (2012)
#123One of the assumptions of vanilla multi-armed bandits is that the underlying reward rates are fixed. It's not valid to assume that in a lot of cases, including e-commerce. The author is dismissive and hard wavy about this and having worked in in e-commerce SaaS I'd be a bit more cautious. Imagine that you are running MAB on an website with a control/treatment variant. After a bit you end up sampling the treatment a l…
> ...the conversion rate for both sides goes up equally.
If the conversion rate "goes up equally", why did you not measure this and use that as a basis for your decisions?
> its aggregate conversion rate goes up faster than the control - you start weighting even more towards that variant.
This sounds simply like using bad math. Wouldn't this kill most experiments that start with 10% for the variant that do not provide 10x the improvement?
Re: Lines of code that beat A/B testing (2012)
#124One of the assumptions of vanilla multi-armed bandits is that the underlying reward rates are fixed. It's not valid to assume that in a lot of cases, including e-commerce. The author is dismissive and hard wavy about this and having worked in in e-commerce SaaS I'd be a bit more cautious. Imagine that you are running MAB on an website with a control/treatment variant. After a bit you end up sampling the treatment a l…
Good point about fluctuating rates for e.g the sales period. But couldn't you then pick a metric that doesn't fluctuate?
Out of curiosity, where did you work? In the same space as you.
Re: Lines of code that beat A/B testing (2012)
#125Earlier quoted context omitted.
Tracks that I’ve primarily seen A/B tests used as a mechanism for gradual rollout rather than pure data-driven experimentation. Basically expose functionality to internal users by default then slowly expand it outwards to early adopters and then increment it to 100% for GA. It’s helpful in continuous delivery setups since you can test and deploy the functionality and move the bottleneck for releasing beyond that.
I wouldn’t call that A/B testing but rather a gradual roll-out.
If you roll it back upon seeing problems, then you're doing something meaningful, at least. IMO 90+% of the value of A/B testing comes from two things, a) forcing engineers to build everything behind flags, and b) making sure features don't crater your metrics before freezing them in and making them much more difficult to remove (both politically and technically).
Re: b), if you've ever gotten into a screaming match with a game designer angry over the removal of their pet feature, you will really appreciate the political cover that having numbers provides...
Re: Lines of code that beat A/B testing (2012)
#126Earlier quoted context omitted.
> I’ve been in companies that have tried dozens if not hundreds of A/B tests with zero statistically significant results. What I've seen in practice is that some places trust their designers' decisions and only deploy A/B tests when competent people disagree, or there's no clear, sound reason to choose one design over another. Surprise surprise, those alternatives almost always test very close to each other! Other pl…
I think trusting your designers is probably the way to go for most teams. Good designers have solid intuitions and design principles for what will increase conversion rates. Many designers will still want a/b tests because they want to be able to justify their impact, but they should probably be denied. For really important projects designers should do small sample size research to validate their designs like we woul…
A/B tests for user interface is very annoying when you are on the phone trying to guide someone how to use a website. "Click the green button on the left" - "What do you mean? There is nothing green on the screen." - "Are you on xyz.com? Can you read out the adress to me please?" ... Oh so many hour wasted in tech support.
Re: Lines of code that beat A/B testing (2012)
#127Pure, disinterested A/B testing where the goal is just to find the good way to do it, and there's enough leverage and traffic that funding that A/B testing is worthwhile is rare. More frequently, A/B testing is a political technology that allows teams to move forward with changes to core, vital services of a site or app. By putting a new change behind an A/B test, the team technically derisks the change, by allowing…
Derisking changes may not work sometimes. For example I don't use Spotify anymore, because of their ridiculous Ab tests. In one month I saw 3 totally different designs of the home and my fav playlists page on my Android phone. That's it. When you open Spotify only when you start your car then it's ridiculous that you can't find anything and you are in a hurry. That was it. I am no longer subscriber and a user of this…
Ever tried a support call helping someone using a website that has A/B testing on? It's a very frustrating experience where you start to think the user on the other side must have mistyped the url. A lot of time wasted on such calls. And yes, the worst is when these things only happen in things you use only when in a hurry.
Re: Lines of code that beat A/B testing (2012)
#128Earlier quoted context omitted.
> I’ve been in companies that have tried dozens if not hundreds of A/B tests with zero statistically significant results. Yea, I've been here too. And in every analytics meeting everyone went "well, we know it's not statistically significant but we'll call it the winner anyway". Every. Single. Time. Such a waste of resources.
Is it a waste? You proved the change wasn't harmful.
To prove that a change isn't harmful is still a hypothesis.
Re: Lines of code that beat A/B testing (2012)
#129Earlier quoted context omitted.
Tracks that I’ve primarily seen A/B tests used as a mechanism for gradual rollout rather than pure data-driven experimentation. Basically expose functionality to internal users by default then slowly expand it outwards to early adopters and then increment it to 100% for GA. It’s helpful in continuous delivery setups since you can test and deploy the functionality and move the bottleneck for releasing beyond that.
I wouldn’t call that A/B testing but rather a gradual roll-out.
I think gradual rollout can use the same mechanism, but for a different readon: avoiding pushing out a potentially buggy product to all users in one sweep.
It becomes an A/B test when you measure user activity to decide whether to roll out to more users.
Re: Lines of code that beat A/B testing (2012)
#130So he compares users to slot machines. OK, I see why the internet sucks now 10 years later.