You'd be shocked at how varied people can be and what effects your actions will have on them. The cumulative effect of those actions can be especially counter-intuitive when there are multiple orders of magnitude at play (like the 100:1 or 1000:1 odds substack must have on anyone interacting positively with the popup).
It wouldn't surprise me at all if any of the following effects (or hundreds of others) are enough to make a difference.
- The people most likely to pay for in-depth tech articles are also high earners with large demands on their time, and they're likely to be interrupted before they finish the article or bounce before the conclusion because they've internalized the meat of the content already. Asking early is annoying, but if it's even 10% as effective then it might overcome the bounce rate. Optimizing total revenue might happen by targeting those people, even at the expense of ruining the sign-in conversion rate.
- Their A/B testing treats sign-ups and subscriptions as information-independent funnels. They're optimizing sign-ups, hoping to therefore optimize revenue, but the sort of person most likely to sign-up after being afronted with a pop-up is also the sort of person who doesn't care that the pop-up appeared before they finished.
- Sign-ups are dominated by people who are hooked on a single article after then first paragraph or three and who can't stand to put it down. After the psychological investment of making an account, they're more likely to stay.
- Sign-ups are roughly orthogonal to the current article. People sign up when they think doing so will have enough value over time, and they have enough signal from the article to figure that out well before they've extracted all the benefit they'll get from the article. The pop-ups only capture people who were about to convert anyway, and the pop-up just made the conversion easier.