Black box optimization simply doesn't work for a/b testing. Product teams should be agile and understand WHY their features are affecting users the way they do.
Why black box A/B testing is BS
leanplum.com