In this case you would not look at how correct people think the horoscopes are, you would look at whether they think their own horoscope is most correct (they don’t know which is which of the ten). The average “subjective correctness” of the horoscopes is not something you would be interested in. You might throw it in somewhere in the paper you are writing, but more to show that ceiling or floor effects are not an issue (i.e. nearly everyone thinks the horoscopes are all either completely correct or completely false and as such you cannot really properly measure – like measuring the distance between New York and Los Angeles with a yard stick or the size of a molecule with a yard stick).
I dislike the outcome this experiment is measuring. I’m not an expert on horoscopes (and wouldn’t want to be one), but to me the primary purported function of horoscopes is to predict the future. As such people might not necessarily be able to say beforehand whether their own horoscope is more correct than that of someone else†. How should they know? But that is easily fixed: Just ask them to rank the horoscopes after the purported predicted time frame …
Altogether this seems like a solid setup to me, though. It would be fun to conduct this, but ultimately pointless. Astrology is just a too low hanging fruit.
(A potential issue might be people’s awareness of stereotypes associated with their astrological sign. I’m not sure how big a role those play of how important and consistent across different astrologers those are – as I said, not an expert – but if stereotypes associated with astrological signs are consistent and generally well known people might be able to identify the correct horoscope just based on the knowledge of those stereotypes, not any predictive power of the horoscopes.)
It was Feynman using the term cargo cult science. And he was not only talking about soft science, he was more generally talking about all of science. And it was not a general condemnation, it was about saying that “cargo cult science” is always a possibility and obviously an issue, but not that it is always an issue. Though I would definitely agree with you in saying that quantitative soft sciences suffer the most from cargo cult thinking.
It’s a paradox, really. “Soft science” is actually really hard. Measuring people and people’s behavior is a nightmare. There are so many variables, so many complex, ever-changing systems. We are at the limits of what we can achieve and our measuring instruments are extremely blunt and imprecise.
Because it’s so hard we are currently not even trying to solve many of the hard problems there. Because we just don’t know how. As such we limit ourselves to simple stuff and the people who are adept at doing the hard stuff (but not as hard as soft science, since that is so hard, no one knows how to properly do it) are doing things that are easier to nail down, with fewer variables – like physics.
And quantitative soft science limits itself by necessity to simple stuff. It simplifies without understanding the underlying complexity. And often slips into cargo cult thinking doing that …
—
† Though I actually think this is the real function of horoscopes: People have to be able to identify with them and think that they fit them when they are reading them. It’s entertainment. That’s why they are full of platitudes and not very specific.