It's not the case. It would be true to say that no one pays full price unless they have a lot of money. Harvard, not unrepresentative of other top U.S. schools in this regard, says that 55% of their students receive financial aid [1]. I infer that 45% are paying the full price of around $80,000 per year [2]. (Some of these 45% are doubtless funded partly by non-Harvard scholarships.)
Playing with Harvard's "Net Price Calculator" [3] makes clear that non-millionaires, let alone non-billionaires, may be asked to pay full price. A lot depends on the number of children, the number in college, and accumulated wealth (excluding retirement accounts and home equity).
Harvard students who receive federal financial aid pay (or their families pay) around $15,500 per year on average [4], which suggests that the distribution of payments may be bimodal: a lot of families pay $80,000 per year, and a lot pay a lot less.
[1] https://college.harvard.edu/financial-aid
[2] https://college.harvard.edu/financial-aid/how-aid-works
[3] https://college.harvard.edu/financial-aid/net-price-calculat...
[4] https://collegescorecard.ed.gov/school/?166027-Harvard-Unive...