You probably don’t need A/B testing
oliverpalmer.com
You probably don’t need A/B testing
1–10 of 11 posts
Re: You probably don’t need A/B testing
#2[deleted]
Re: You probably don’t need A/B testing
#3Lets fix the title - if you think you need A/B testing, you don't. I have explained this to small organizations that are running sites getting tens of thousands of hits and just completely floundered at it.
The people whom A/B testing can help are running sites with million of hits.
Re: You probably don’t need A/B testing
#4Lets fix the title - if you think you need A/B testing, you don't. I have explained this to small organizations that are running sites getting tens of thousands of hits and just completely floundered at it. The people whom A/B testing can help are running sites with million of hits.
Haha, I like your title but I don't think it's accurate.
Many, many sites with millions of visitors are not yet A/B testing. Vastly more than you would think.
Re: You probably don’t need A/B testing
#5Lets fix the title - if you think you need A/B testing, you don't. I have explained this to small organizations that are running sites getting tens of thousands of hits and just completely floundered at it. The people whom A/B testing can help are running sites with million of hits.
Haha, I like your title but I don't think it's accurate. Many, many sites with millions of visitors are not yet A/B testing. Vastly more than you would think.
Ok, lets take it a step further ;)
The sites with millions of hits not A/B testing might be helped by an A/B test but more likely will just chase ghosts and give it up.
Or they could just hire a good UI/UX and conversion expert and get the insights for free.
Take agriculture - you are running a large farm which is hundreds to thousands of acres. Do you run experiments with soil, seeds, fertilizer, crop rotation schemes to increase yield? No, that would be silly, take forever and make you poorer and perhaps no wiser. Instead, you hire a "expert" who has done it before and knows the "best practices".
Re: You probably don’t need A/B testing
#6Lets fix the title - if you think you need A/B testing, you don't. I have explained this to small organizations that are running sites getting tens of thousands of hits and just completely floundered at it. The people whom A/B testing can help are running sites with million of hits.
Strongly disagree - A/B testing can be heavily used in e-commerce and online shops. You don't need "millions" of hits in order to test which products do better on the homepage, which discounts encourage more people to click buy, etc.
Re: You probably don’t need A/B testing
#7Earlier quoted context omitted.
Haha, I like your title but I don't think it's accurate. Many, many sites with millions of visitors are not yet A/B testing. Vastly more than you would think.
Ok, lets take it a step further ;) The sites with millions of hits not A/B testing might be helped by an A/B test but more likely will just chase ghosts and give it up. Or they could just hire a good UI/UX and conversion expert and get the insights for free. Take agriculture - you are running a large farm which is hundreds to thousands of acres. Do you run experiments with soil, seeds, fertilizer, crop rotation schem…
Very interesting analogy but not a good equivalence IMO.
Soil and rotation are pretty standard in that the best permutations are already known, unless you want to experiment something new.
Websites do not have such known standards- a crappy looking amazon website works better than most UI experts would suggest.
Re: You probably don’t need A/B testing
#8Lets fix the title - if you think you need A/B testing, you don't. I have explained this to small organizations that are running sites getting tens of thousands of hits and just completely floundered at it. The people whom A/B testing can help are running sites with million of hits.
UX research can act as observations to form an initial hypothesis, while A/B testing can acquire the necessary data to perform experiments in way that assimilates the scientific method.
It might be a scale issue for sure, but qualitative data alone is not enough, especially when dealing with user biases and small, non representative populations.
Re: You probably don’t need A/B testing
#9Lets fix the title - if you think you need A/B testing, you don't. I have explained this to small organizations that are running sites getting tens of thousands of hits and just completely floundered at it. The people whom A/B testing can help are running sites with million of hits.
Disagree - thousands of hits can provide huge amounts of insightful data to make A/B testing valuable and productive when its captured and analyse effectively as part of a broader UX and Design System strategy. If you don’t have those it doesn’t matter what volume of data you have.
Using data to test product iteractions, interface changes and catch low hanging fruit is totally standard and widely practiced. Companies not willing to use data driven measurement are lagging behind by at least 5 years.
Re: You probably don’t need A/B testing
#10Lets fix the title - if you think you need A/B testing, you don't. I have explained this to small organizations that are running sites getting tens of thousands of hits and just completely floundered at it. The people whom A/B testing can help are running sites with million of hits.
UX research can act as observations to form an initial hypothesis, while A/B testing can acquire the necessary data to perform experiments in way that assimilates the scientific method. It might be a scale issue for sure, but qualitative data alone is not enough, especially when dealing with user biases and small, non representative populations.
Not enough for what?