Earlier quoted context omitted.
I've noticed the same thing with LP's. While LP's are technically inferior to red book CD's, they're often mastered better.
Why is this? I only have 16 bit 44.1kHz "normal" CDs and I notice some terrible mastering jobs, eg. RHCP Stadium Arcadium loses the snare for the first track and is like listening to minutes of distortion yet apparently the vinyl was really well mastered. Is there a technical reason why the mastering is so different for the two mediums, CD versus vinyl?
Thus, people started mastering CDs for loudness.
An alternative idea of mine is simpler. Loud music is considered worse than quiet music (quiet music sounds worse, but loud music still does, and also bothers other people). So, when you need to pick a volume setting for your collection, you bias towards setting it lower, so the really loud ones don't become too loud. Thus, the quieter CDs are annoying because they always sound quiet, whilst the load CDs sound about right, because your volume is much more suitable for them.