Earlier quoted context omitted.
> Didn’t they have a way to release changes and measure impact before just forcing it on their entire user base? Many possible changes have positive effects short term, but negative or even disastrous effects long term. A/B tests won't ever run long enough to discover these long term costs. When A/B facts displace common sense instead of it complementing it, you can a/b yourself into your own grave.
If you run an experiment and time spent per day is cut in half, that's a pretty clear signal the experiment shouldn't ship, vision or not.
It's generally assumed that with advertising, more time == more money, but even that's somewhat questionable.
In a B2B context, time spent per day cut in half sounds like a huge productivity boost. Of course, it could also be that you've made it so difficult to use now that people do a little bit, export to CSV and then use some other tool. But the data won't tell you that.