Earlier quoted context omitted.
There is a chance of survivorship bias. You get a bunch of messages through the chat interface and all of a sudden you believe it adds value, when in reality you're potentially annoying a whole bunch of users that aren't necessarily telling you that they're annoyed. A favourite story[1] (and I can't remember where I read it first) is the WWII bombers that the US military were trying to understand how to protect by an…
The organization can’t do anything about visitors who are testing them with secret little rules about when they will withhold business. More typical power users are able to just ignore support options they don’t want to use and use the product. I like the WW2 story as well, but when chat increases total interactions and conversions, it’s hard to make a case for survivorship bias.
For example, chat popups that ask you to interact before you've had a chance to even read the title of the page.
Every customer who hasn't purchased yet is part of a funnel, and you absolutely CAN test with an A/B test (or equivalent) whether you're losing people when you annoy them by popping something up on the screen.
And perhaps I'm an outlier. If the A/B test says you get more sales when you annoy the customer, who am I to argue (or it annoys me and not them). But it does mean you will have them as a customer and not me, and that's fine.
I don't think I'm wrong to use this as one of many indicators of how annoying a SaaS is going to be to deal with over the long term.