Google Optimize now free for everyone

Some pretty easy rules of thumb, assuming you have a decent grasp on your economics. Look at it as a "low hanging fruit" optimization problem -- do you put resources into running a test (+ opportunity cost for lost sales), or into something else?

Suppose you have 10k monthly sessions with a 0.5% conversion rate (50 conversions). How many more customers would you need in order to prioritize running a test? If 55 conversions in a given month means you crush important KPIs, then that's probably worth testing -- you just need a 10% lift.*

Also keep in mind that running A/B tests (1 control, 1 treatment) is suboptimal. That tests, "does this beat what I have now?" The more important question is "what is my best option?".

OTOH, if other things like messaging and product are stable, you can test a smaller traffic site by leaving it running longer.

My rough estimate is 100 conversion events in the time the test runs. So if I have 100 conversion events in 1 month, it may make sense to run a 2-3 option + 1 control test for 1 month.

(You can also test much larger things than buttons. For startups, I like to suggest trying out positioning or value statements and seeing how visitors respond!)

* however, it'll take a long time for you to reach statistical confidence for a 10% lift in rate, with only 50 conversion events across all tests.

Google Optimize offers MAB testing, which is awesome. Personally, I don't see it as a "when do results become reliable" problem -- it's a "what is the cost of a false positive, false negative, or 'do nothing' decision?"

At least thinking about it is important NOW, regardless of what stage your startup is. It can be extremely difficult to try to slot a 3rd party A/B testing solution into your product (or really hard to roll your own) if your infra doesn't support it from the start. Also, hire a data scientist! (Disclaimers: I am not a data scientist, I just think everyone needs one! I have worked on an experimentation system at $BIG_COMPANY.)

I'd suggest thinking about the following BEFORE YOU RUN A SINGLE A/B TEST:

1) Key Metrics: Define these. They are the general, "I don't care what your experiment is about, these numbers are important." Every experiment you run should automatically track these metrics. You should also give the ability to define custom metrics, since an experiment that changes some random button color probably wants to look at how many people clicked the button, which is almost definitely NOT a key metric.

2) Logging Infrastructure: Make sure that you have a easy-to-use, reliable data pipeline set up for logging and processing events. Bad logging == bad experiment results. Also consider streaming vs batch processing for updating experiment results.

3) Population Management: How do your experiments segment users? Are variants calculated in realtime? Batched with some SLA for lag? Are they sticky?

4) Mutual Exclusion: People running experiments often want "their" users excluded from other experiments.

5) Guardrails: Do your experiments automatically shut off if there is a catastrophic decline in one or more key metrics? What safety measures do you have around determining if an experiment is safe/valid? How do you handle cleaning up data when there's a problem? What sorts of actions invalidate an experiement's existing results? Does your entire site break if your A/B Testing service is down for whatever reason?

6) Cleanup/Ownership: Experiments don't run forever (at least they shouldn't!). Cleaning up old features, populations, etc can a pain, especially when the people that wrote the stuff originally no longer work at the company. Make cleanup mandatory and as easy as possible.

There's a lot more, but I'm tired now. A/B testing is complex. There are lots of resources out there, though. Look for white papers on the subject, they're surprisingly approachable. Example from Microsoft: https://exp-platform.com/Documents/2017-08%20KDDMetricInterp...

There is no such requirement, and I thought I was clear that I appreciated the post, and was merely trying to add the information.

Also: the headline has "now" in it. It's not the end of the world, but I've seen correction notes on NYT etc articles for far smaller matters. It's just a general principle that true information > false information.

> No one scrolls to the end of a long form article to get a recipe.

I highly doubt this is true. Scrolling to the bottom of a page is a very low cost to pay for a recipe.