Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Conservation of Intent: why A/B tests aren’t as effective as they look

andrewchen.co

Post reply on HN
Print view View original

41–50 of 59 posts

Re: Conservation of Intent: why A/B tests aren’t as effective as they look

#41

Post by draw_down »

birken wrote: post #3

I could not disagree with this more. I remember vividly having this "low-intent" vs "high-intent" debate at Thumbtack, when we rolled out changes that A/B tests showed increased conversion (by a lot), but some people in the company thought the changes were ugly and "off-brand" and argued they brought in the wrong type of customers. So we ran the test again that we knew raised conversion by a lot, and then followed th…

Hmm, sounds like A/B tests are really good, assuming you run them in a way that almost nobody runs them in practice.

Re: Conservation of Intent: why A/B tests aren’t as effective as they look

#42

Post by s-shellfish »

swied wrote: post #27
snovv_crash wrote: post #17

A/B tests tell you about short term gains, but don't tell you about long term issues you may be accumulating due to things like dark patterns, clickbait headlines, shoddy article topics and more. A/B tests don't take into account the loss of prestige or reputation that the options give. I've seen this repeatedly with ArsTechnica, which has devolved into so much political and clickbait material that I don't even reall…

So true about the danger of focusing on short term gains. I think that it is a problem of picking the wrong things to optimize and then sampling on the wrong entities. Conversion rate is often the most commonly used target, but, it is short sighted. In the big picture a company is really just trying to maximize profit. Note that you can't use traditional A/B testing tools on a value like total profit. You need an ave…

User experience analysis is always going to require a human mind to synthesize the material. The problem is that automation eventually gets noticed by users, thus automation implies to a user 'the programmers/whoever believes my mind and preferences can be automated'.

That's always going to throw a wrench into automation. So yes, it's hard work that requires real thinking with a clear focus on the users and how the users feel about the software. Software needs to care about users if it is dependent on users for it's business model!

Re: Conservation of Intent: why A/B tests aren’t as effective as they look

#43

Post by s-shellfish »

citrablue wrote: post #33
snovv_crash wrote: post #17

A/B tests tell you about short term gains, but don't tell you about long term issues you may be accumulating due to things like dark patterns, clickbait headlines, shoddy article topics and more. A/B tests don't take into account the loss of prestige or reputation that the options give. I've seen this repeatedly with ArsTechnica, which has devolved into so much political and clickbait material that I don't even reall…

What's the alternative to A/B testing? Everywhere I've worked, design teams hate A/B testing, but suggest "trust us we know what we're doing" as the alternative. A/B/n testing let's you explore a search space. There are issues that can come up from that, but IMO it's a better option than this Dilbert comic[0]. [0] http://dilbert.com/strip/2014-10-27

Listen to people about their complaints and preferences as intellectual equals.

Re: Conservation of Intent: why A/B tests aren’t as effective as they look

#44

Post by mygo »

snovv_crash wrote: post #17

A/B tests tell you about short term gains, but don't tell you about long term issues you may be accumulating due to things like dark patterns, clickbait headlines, shoddy article topics and more. A/B tests don't take into account the loss of prestige or reputation that the options give. I've seen this repeatedly with ArsTechnica, which has devolved into so much political and clickbait material that I don't even reall…

This has less to do with the use of A/B testing than it does the experimental design.

What are they testing for in their metrics? Is it just click rates? Well that’s a patently losing strategy in the long term.

What if whether or not you came back over time is what was being measured? Would that break some cardinal rule that all A/B tests must be done within a short time period?

Researchers has been doing A/B testing for centuries. Till now they’ve just called it having a control variable and an independent variable. Experimental design is a big deal, and something that is taught to every graduate student researcher, but I don’t see the tech blogs giving experimental design enough focus — at least not in their click-bait headlines. Sometimes people don’t design good experiments and then blame the methods.

Re: Conservation of Intent: why A/B tests aren’t as effective as they look

#45

Post by tehlike »

kolpa wrote: post #39
tehlike wrote: post #32

Earlier quoted context omitted.

Not entirely true. You can setup your experiment to be user keyed (as it should be) and can run it for a long time, or can run the hold back for a long time Something as basic as mod(hash(experiment_group_id or layer id, userid), 1000) would give you a user stable experiment distribution that you can analzy long term impact on.

Long-term A/B testing has major problems in the modern world where users can communicate with each other (both onsite and out of band) and can use multiple accounts / devices / browsers

communicating user is less of a concern, cross device _is_ a real concern. In some cases it's minimized with user logins, though.

Re: Conservation of Intent: why A/B tests aren’t as effective as they look

#46

Post by citrablue »

s-shellfish wrote: post #43
citrablue wrote: post #33

Earlier quoted context omitted.

What's the alternative to A/B testing? Everywhere I've worked, design teams hate A/B testing, but suggest "trust us we know what we're doing" as the alternative. A/B/n testing let's you explore a search space. There are issues that can come up from that, but IMO it's a better option than this Dilbert comic[0]. [0] http://dilbert.com/strip/2014-10-27

Listen to people about their complaints and preferences as intellectual equals.

Sure, but not everyone's opinion is equally valid. I've worked with a number of design teams who confuse their opinion (an educated opinion, which I respect) as a fact. Facts beat opinions -- if a short-term optimization will be a long term loser, then prove it.

Unfortunately, it's not really possible to prove/disprove an opinion on what a design element will do.

Additionally, optimization testing allows you to try out different options. Going with someone's (educated) gut feel is a worse decision than exploring the option space.

Ideally, you'd test everything. In the real world, resource constraints apply, so at issue is how to establish an internal discipline where facts trump opinions, and opinions can be tested then either discarded or confirmed.

Re: Conservation of Intent: why A/B tests aren’t as effective as they look

#47

Post by Franciscouzo »

citrablue wrote: post #33
snovv_crash wrote: post #17

A/B tests tell you about short term gains, but don't tell you about long term issues you may be accumulating due to things like dark patterns, clickbait headlines, shoddy article topics and more. A/B tests don't take into account the loss of prestige or reputation that the options give. I've seen this repeatedly with ArsTechnica, which has devolved into so much political and clickbait material that I don't even reall…

What's the alternative to A/B testing? Everywhere I've worked, design teams hate A/B testing, but suggest "trust us we know what we're doing" as the alternative. A/B/n testing let's you explore a search space. There are issues that can come up from that, but IMO it's a better option than this Dilbert comic[0]. [0] http://dilbert.com/strip/2014-10-27

I think a better approach might be using bayesian optimization, each variation would be a dimension, and the algorithm would find the (almost) optimal possibility instead of a/b testing, which finds the local maximus.

Re: Conservation of Intent: why A/B tests aren’t as effective as they look

#48

Post by citrablue »

swied wrote: post #35
citrablue wrote: post #33

Earlier quoted context omitted.

What's the alternative to A/B testing? Everywhere I've worked, design teams hate A/B testing, but suggest "trust us we know what we're doing" as the alternative. A/B/n testing let's you explore a search space. There are issues that can come up from that, but IMO it's a better option than this Dilbert comic[0]. [0] http://dilbert.com/strip/2014-10-27

A/B testing is a great tool for incrementally improving upon a product design given the correct business strategy. Just don't let the limitations of A/B testing drive how you choose your business strategy. The strategy is ultimately about maximizing total profit, and not necessarily short term profit per user.

How do you create a discipline around making good decisions for business strategy? I've seen "highest paid person" win, I've seen "Most vocal" win, I've seen "5 versus 1" win.

The testing advocates I've worked with have an attitude of "I don't know what will work - that's why we're testing". I have not seem much of that attitude from most other individuals/teams involved in decision making -- they have preferred to say "this will work", and get angry when I say, "compared to what? how do you know?"

edit - fwiw, I totally agree with understanding the limitations of testing. You have to know what it can be good for. My argument is that it's actually better for long term strategy than most other decision making processes I have experienced, which usually boils down to "gut based".

Re: Conservation of Intent: why A/B tests aren’t as effective as they look

#49

Post by s-shellfish »

citrablue wrote: post #46
s-shellfish wrote: post #43

Earlier quoted context omitted.

Listen to people about their complaints and preferences as intellectual equals.

Sure, but not everyone's opinion is equally valid. I've worked with a number of design teams who confuse their opinion (an educated opinion, which I respect) as a fact. Facts beat opinions -- if a short-term optimization will be a long term loser, then prove it. Unfortunately, it's not really possible to prove/disprove an opinion on what a design element will do. Additionally, optimization testing allows you to try o…

I mean the users of software. Instead of trying to predict what is the best design, create interfaces that make it easy to aggregate that data, and take that information at face value.

When it comes to design, there often aren't facts (one may even suggest facts don't exist when it comes to design). Study the history of art to contest that. Otherwise, listen to users like they are humans, and see whether it matches with the designers advice. Simple.

Re: Conservation of Intent: why A/B tests aren’t as effective as they look

#50

Post by RA_Fisher »

mygo wrote: post #44
snovv_crash wrote: post #17

A/B tests tell you about short term gains, but don't tell you about long term issues you may be accumulating due to things like dark patterns, clickbait headlines, shoddy article topics and more. A/B tests don't take into account the loss of prestige or reputation that the options give. I've seen this repeatedly with ArsTechnica, which has devolved into so much political and clickbait material that I don't even reall…

This has less to do with the use of A/B testing than it does the experimental design. What are they testing for in their metrics? Is it just click rates? Well that’s a patently losing strategy in the long term. What if whether or not you came back over time is what was being measured? Would that break some cardinal rule that all A/B tests must be done within a short time period? Researchers has been doing A/B testing…

Bingo! "A/B testing" aka completely randomized design: https://www.itl.nist.gov/div898/handbook/pri/section3/pri331...

Mountains can be moved with randomized control trials.

Post reply on HN