Well, mathematics is hard. It takes a lot of effort to really learn it, a lot of dedication and (self-)motivation. Not everyone can do it, and, frankly, very few people - even among those who have spent many years studying mathematics - could say that it is the best way to live your life.
I profoundly disagree. I believe that mathematics can be learned and appreciated by many, because mathematics is a language for articulating inner urges and perceptions that are common to a wide range of creatures, not only humans. This idea that mathematics is only for the few is self fulfilling, and perpetuated by narrow ideas of what it constitutes.
I do agree that math can and should be widely appreciated.