I read the comments first, and then eagerly read your post - because I hate Facebook and wanted to scorn you, but having read it I see that it's not really Facebook-centric at all, containing only a glancing reference.
That said, I do think there are problems with your use of statistics and you kind of do need to consider the numbers in absolute terms. If I check my idea out with ten people I know in my target demographic, and five of them like it, I probably shouldn't expect 50% of the target market will be interested in my idea. My sample size is too small and also not really representative of the demographic because there is a selection bias at play - they're people I know, so maybe I know the only five people in the world who have the same problem I'm trying to sovle.
Likewise, in your example, you talk about showing a concept to people on a Facebook social group, but people on a Facebook social group aren't really representative of your target demographic either. "Only 20 bucks a month? Sounds great, I love interacting with apps!" may be something a Facebook engineer would say, but that may be less likely from the mouth of the average 25-44 year old.
I do think testing your ideas out and seeing how users respond to them is a good practice but there is inherently danger to doing tests and coming up with statistics if you believe your own tests. You might run the experiments you describe and feel good about your numbers and invest time and money to go to a wider audience only to find out that a lot of people think 20 dollars a month is too expensive for an app that gives workout advice (which, I predict, I could find on the current app store for free) or that they don't like working out at home or from an app, etc.
If you know you don't know what's ahead you'll proceed cautiously. If you falsely believe you know what is ahead due to biased statistics or insufficient sample sizes, you may blunder painfully.
I also caught a few references to Thiel's "Zero to one" in there, but I feel the approach you're describing, at least the hypothetical example, is somewhat contradictory to the advice in that book. I'd characterize your approach as:
1. figure out the market - (Big number, everyone interested in fitness)
2. guess what piece you can serve - (small number - only 2% need to try!)
3. figure out if you can serve them - (small number - maybe 10% retention)
In other words, "We just need a small piece of this big market!"
From Zero to One:
>Now I think the opposite version of this is always where you have super big markets and there is so many different things that went wrong with all the clean tech companies in the last decade. But the one theme than ran through almost all of them is that they all started with massive markets. Every clean tech powerpoint presentation that one saw in the years 2005 to 2008, which is the clean tech bubble in Silicon Valley, started with where the energy market, where the market was measured in hundreds and billions of trillions of dollars. And then once you're sort of like a minnow in a vast ocean, that is not a good place to be. That means you have tons of competitors and you don’t even know who all the competitors are.
>You want to be a one of a kind company. You want to be the only player in a small ecosystem. You don’t want to be the fourth online pet food company. You don’t want to be the tenth solar panel company. You don’t want to be the hundredth restaurant in Palo Alto. Your restaurant industry is a trillion dollar industry. So if you do a market size analysis, you conclude restaurants are fantastic business to go into. And often large existing markets typically means that you have tons of competitions so it's very very hard to differentiate. The first very counterintuitive idea is to go after small markets, markets that are so small people often don't even think that they make sense. That's where you get a foothold and then if those markets are able expand, you can scale into a big monopoly business.
In other words, if I looked at "2% of people who saw my Facebook post left their email" I'd probably think the same thing I'd think prior to doing any experiment, which is "There is a lot of fitness stuff, including apps and gyms, this probably won't work". But, I'm also a pessimist, so maybe that's just me.
Anywho - sorry if I've been too negative. I did enjoy reading your post, and I hope you hit the target numbers you want on your substack subscriptions from this post! :)