> Intuitively, how could it be correct that people pay thousands of dollars on these courses and receive zero or near to zero benefit from them?
Because the people paying have a sample size of n=1?
Look, any test will be subject to random measurement noise. If you got a bad night's sleep, got lost on the way to the test, had an argument with your parents or broke up with your boy/girlfriend, you're probably going to do worse. Conversely, if you had a nutritious breakfast or take a version of the test with lots of questions similar to the ones you practiced on you'll do better. Some people get "lucky" (for lack of a better word), while some get "unlucky".
The thing is, unlucky people are far more likely to retake the test and pay for test prep. So people enrolled in you test prep class are NOT a random sample of high school students. The sample is biased to include more unlucky people and fewer lucky people.
People paying for your service don't know whether their child was unlucky or not.
I used the work "lucky" for a reason. It is less likely that those negative factors would affect the student both times they take the test than just one time. So someone who encountered a confluence of unlucky factor for the first test might score higher on the second test simply by having fewer unlucky factors--or even just being more familiar with the test, having taking it previously.
> If this were true no one would take the courses, the courses would be rated poorly, and the underlying business would fail.
Your course doesn't need to actually help anyone. Your customers just need to believe it helped them. How do they know whether you actually helped, or whether the improved test scores are down to luck?
Does your average customer have 20 grandchildren who they can randomly split into test and control groups? Or do they have an imperfect understanding of statistics?