11–20 of 75 posts
Maybe I’m missing something. The goal of an A/B test is to test a hypothesis. Where that hypothesis comes from is irrelevant. Sure you can waste your time testing stupid hypotheses that don’t have a lot of business impact, but that’s beside the point.
Data may not tell what to do next, but models do. For example Bayesian optimisation framework tries to do exactly that.
I've never worked at a company where we were anywhere close to being able to A/B testing on customers. We just always shot from the hip. I think if you are doing that kind of testing may e you've run out of helpful ideas? Talking to real customers and helping them solve real problems is really potent. And you can get more than just the color of a button. You can get the direction your company needs to go for months.…
Then you have people that try to get greedy. On more than one occasion I have designed a test where two variables change, results are great, rollout projections are great, the stakeholder attempts to do the rollout without changing the variable that creates incremental expense, and the rollout does not meet projections. Then they reluctantly do what they were supposed to do in the first place and everything is fine.
I've never worked at a company where we were anywhere close to being able to A/B testing on customers. We just always shot from the hip. I think if you are doing that kind of testing may e you've run out of helpful ideas? Talking to real customers and helping them solve real problems is really potent. And you can get more than just the color of a button. You can get the direction your company needs to go for months.…
How much faster can we process payments through provider A versus B in different countries around the world?
If we offer insurance after checkout, do we convert more than offering it before?
What ranking algorithm of skus leads to the highest conversion?
Data may not tell what to do next, but models do. For example Bayesian optimisation framework tries to do exactly that.
And as the saying goes: all models are wrong (but some are useful). You're going to have to sit down and do the thinking eventually anyway. May as well start by incorporating it into your process from the beginning.
> Intuition is underrated > Spend time where your customers are and make your own conclusions. This is a great article, very well-written, and I enjoyed reading it. However, could intuition and spending time with customers be considered another way of collecting data points to inform data-driven decisions?"
Generally when talking about data driven you are talking about quantitative data and not qualitative data.
I've never worked at a company where we were anywhere close to being able to A/B testing on customers. We just always shot from the hip. I think if you are doing that kind of testing may e you've run out of helpful ideas? Talking to real customers and helping them solve real problems is really potent. And you can get more than just the color of a button. You can get the direction your company needs to go for months.…
Much of A/B testing is far deeper than the color of buttons. How much faster can we process payments through provider A versus B in different countries around the world? If we offer insurance after checkout, do we convert more than offering it before? What ranking algorithm of skus leads to the highest conversion?
Does a customer care if you shave off 2% of the final cost or do they care about having world-class customer support?
Does a customer care if a product has a higher conversion rates or if it is the product they were looking for in the first place?
Does a customer care more about how long payment processing takes or do they care that it takes their local mobile payment app?
The only way to know is to talk to customers. Without doing so you’re just coloring a different variety of button.
I've never worked at a company where we were anywhere close to being able to A/B testing on customers. We just always shot from the hip. I think if you are doing that kind of testing may e you've run out of helpful ideas? Talking to real customers and helping them solve real problems is really potent. And you can get more than just the color of a button. You can get the direction your company needs to go for months.…
One of the many issues is you only get tot talk to customers willing to honestly talk to you.
That means you can't hear from potential new customers you wouldn't know were part of your market. You also don't hear from customers who would want to leave you but just haven't put it into words yet.
A/B testing helps get more insight into what customers actually do (and not what they tell you) and also get numbers on how big of an impact your changes have. The time to wait for the results is insignificant compared to the impact of ill changes in general.