Whom the gods would destroy, they first give real-time analytics (2013)

Advice appreciated but I'm exceedingly familiar with experimental design haha, what I understand far less well though is the integration of the toolset into a business/product development context. I can see how having a staggered cadence of stopping, reflecting on the experimental design, and making a decision is wise. But it still seems that you could perform the experiment using MAB to keep the profit motive happy (you don't want to waste potential click-throughs just because you like p-values, maybe tune it to be more conservative about shifting heavily to one arm) and then have some period where you stop the experiment to pause and reflect.

Heck you don't even have to do MAB if you don't want to, just don't use NHST. The Bayesian "flavor" of NHST (credible intervals around posterior expected values) has absolutely no problem with optional stopping. Run the experiment until you've got a precise enough estimate, then sit back and make your product decisions.

I guess where I'm going with all this is that it seems like the post's strongest point is "good product decisions require time, and realtime analytics bamboozle us into thinking fast decisions are better". All the stuff about NHST seems kind of tangential. Looking at it again I see that it's like a decade old, so I think this is the best explanation for why they were targeting NHST more aggressively. I would hope in our post-replication crisis world (hopefully "post", anyways) data scientists and A/B testers are more prudent about some of these better-known pitfalls.

I agree with you.

This is why I think the idea of technocracy or "evidence based politics" is ultimately a mirage. Sure you can maybe assess some policy but the metrics you're choosing to measure or optimise for are political by their very nature. One's evidence based policy isn't the same as mine.

Health-outcomes-wise it would be better to force everyone to eat salad or whatever but that's only one dimension to optimise on at the expense of freedom and life enjoyment.

Tying it back to tech maybe going down market improves your conversion and lowers your CAC but maybe you've just acquired a bunch of customers with low value, high churn and high costs.

Maybe the sales of Amazon Prime are showing gangbusters returns with the dark patterns but now people loathe your brand and are hoping to see you hit with an FTC banhammer.

There's no silver bullet to this stuff, sure you should probably measure it but ultimately you have to make a decision and be guided by gut instinct and beliefs.

I think business decisions map very well onto the binary decisions implied by NHST A/B testing, which is partly why we put so many resources into studying those problems in the early 20th century.