One of my coworkers is a trained particle physicist and he informs me he almost never sees properly designed experiments used by our A/B testers. The result is that the testers almost always find what they are looking to find.
Conservation of Intent: why A/B tests aren’t as effective as they look
31–40 of 59 posts
Re: Conservation of Intent: why A/B tests aren’t as effective as they look
#32A/B tests tell you about short term gains, but don't tell you about long term issues you may be accumulating due to things like dark patterns, clickbait headlines, shoddy article topics and more. A/B tests don't take into account the loss of prestige or reputation that the options give. I've seen this repeatedly with ArsTechnica, which has devolved into so much political and clickbait material that I don't even reall…
Not entirely true. You can setup your experiment to be user keyed (as it should be) and can run it for a long time, or can run the hold back for a long time
Something as basic as mod(hash(experiment_group_id or layer id, userid), 1000) would give you a user stable experiment distribution that you can analzy long term impact on.
Re: Conservation of Intent: why A/B tests aren’t as effective as they look
#33A/B tests tell you about short term gains, but don't tell you about long term issues you may be accumulating due to things like dark patterns, clickbait headlines, shoddy article topics and more. A/B tests don't take into account the loss of prestige or reputation that the options give. I've seen this repeatedly with ArsTechnica, which has devolved into so much political and clickbait material that I don't even reall…
What's the alternative to A/B testing? Everywhere I've worked, design teams hate A/B testing, but suggest "trust us we know what we're doing" as the alternative.
A/B/n testing let's you explore a search space. There are issues that can come up from that, but IMO it's a better option than this Dilbert comic[0].
Re: Conservation of Intent: why A/B tests aren’t as effective as they look
#34Earlier quoted context omitted.
> Let's people check test too often and allows them to stop the test when it hits a good result I admit to attempting to be guilty of this in the past and being stopped by our analytics team (in the sense that they took the time to patiently explain to me why what I was doing was statistically unsound). It's not obvious, IMO.
Meh. There is statistics for the purpose of uncovering Truth, and statistics for the purpose of making a business decision. The difference is that when we talk about Truth, a small error is still an error. When we make business decisions, it is fine to make a decision that is probably right, and we know isn't far wrong. Here is a perfectly valid test procedure that illustrates the difference. Decide the most time you…
I mean I think the only reason it would horrify your analytics team is that if you want to do something that sophisticated, you may as well just use a better multi-armed bandit function like Thompson Sampling or UCB-1 (which is very, very similar to what you've described, although more formalized).
So I think its wrong to say that you'd never find it in a stats class.
Re: Conservation of Intent: why A/B tests aren’t as effective as they look
#35A/B tests tell you about short term gains, but don't tell you about long term issues you may be accumulating due to things like dark patterns, clickbait headlines, shoddy article topics and more. A/B tests don't take into account the loss of prestige or reputation that the options give. I've seen this repeatedly with ArsTechnica, which has devolved into so much political and clickbait material that I don't even reall…
What's the alternative to A/B testing? Everywhere I've worked, design teams hate A/B testing, but suggest "trust us we know what we're doing" as the alternative. A/B/n testing let's you explore a search space. There are issues that can come up from that, but IMO it's a better option than this Dilbert comic[0]. [0] http://dilbert.com/strip/2014-10-27
A/B testing is a great tool for incrementally improving upon a product design given the correct business strategy. Just don't let the limitations of A/B testing drive how you choose your business strategy. The strategy is ultimately about maximizing total profit, and not necessarily short term profit per user.
Re: Conservation of Intent: why A/B tests aren’t as effective as they look
#36> You ship an experiment that’s +10% in your conversion funnel. Then your revenue/installs/whatever goes up by +10% right? Wrong :( Turns out usually it goes up a little bit, or maybe not at all.
Never mind "The difference between high- and low-intent users", this could be explained in terms of regression-toward-the-mean, a phenomenon mention in neither the article, nor the discussion here.
Have 1000 students do an IQ test. Pick the top 20 students. Have them do another IQ test next week. Their mean score second time round will almost certainly be lower than their mean score first time round. The reason they made the top 20 the first time round was a combination of having a high true IQ, and being lucky on the day. Second time round, they aren't 'defined to be lucky', as it were.
It's the reason movie sequels tend to be worse than the original. The reason the sequel was made was that the original movie was far more successful than the average movie, on account of both unusually skillful creators, and unusually good luck. Second time round, you can't count on the luck component again.
Re: Conservation of Intent: why A/B tests aren’t as effective as they look
#37A/B tests tell you about short term gains, but don't tell you about long term issues you may be accumulating due to things like dark patterns, clickbait headlines, shoddy article topics and more. A/B tests don't take into account the loss of prestige or reputation that the options give. I've seen this repeatedly with ArsTechnica, which has devolved into so much political and clickbait material that I don't even reall…
The last point you mentioned is why I stopped visiting Ars. I can even live with the articles themselves being slanted, but the comments section seemed even more heavily slanted.
Re: Conservation of Intent: why A/B tests aren’t as effective as they look
#38A/B tests tell you about short term gains, but don't tell you about long term issues you may be accumulating due to things like dark patterns, clickbait headlines, shoddy article topics and more. A/B tests don't take into account the loss of prestige or reputation that the options give. I've seen this repeatedly with ArsTechnica, which has devolved into so much political and clickbait material that I don't even reall…
> I've seen this repeatedly with ArsTechnica Can you expand? I am an Ars reader, and I too find it frustrating compared to what it used to be. I wish there were more in-depth technical articles; I find it too light on details, written for a non-tech audience. I really, really wish it had solid technical content, since it's what got me reading it. But I don't characterise it as click-bait (it seems clear) nor as espec…
Politics and a Climate and Freedom are interesting, but not what makes Ars Technica interesting.
Re: Conservation of Intent: why A/B tests aren’t as effective as they look
#39A/B tests tell you about short term gains, but don't tell you about long term issues you may be accumulating due to things like dark patterns, clickbait headlines, shoddy article topics and more. A/B tests don't take into account the loss of prestige or reputation that the options give. I've seen this repeatedly with ArsTechnica, which has devolved into so much political and clickbait material that I don't even reall…
Not entirely true. You can setup your experiment to be user keyed (as it should be) and can run it for a long time, or can run the hold back for a long time Something as basic as mod(hash(experiment_group_id or layer id, userid), 1000) would give you a user stable experiment distribution that you can analzy long term impact on.
Long-term A/B testing has major problems in the modern world where users can communicate with each other (both onsite and out of band) and can use multiple accounts / devices / browsers
Re: Conservation of Intent: why A/B tests aren’t as effective as they look
#40Earlier quoted context omitted.
> Let's people check test too often and allows them to stop the test when it hits a good result I admit to attempting to be guilty of this in the past and being stopped by our analytics team (in the sense that they took the time to patiently explain to me why what I was doing was statistically unsound). It's not obvious, IMO.
Meh. There is statistics for the purpose of uncovering Truth, and statistics for the purpose of making a business decision. The difference is that when we talk about Truth, a small error is still an error. When we make business decisions, it is fine to make a decision that is probably right, and we know isn't far wrong. Here is a perfectly valid test procedure that illustrates the difference. Decide the most time you…
Truth isn't that different from business. Both are statistical. Making a decision that is not statistically valid is bad business, as you could be making things worth as often as better.