I think a lot of it is that people mostly think of it as "the middle phase of adult life" rather than "the middle of your entire life"; 20s and 30s are when people traditionally establish their careers and families, 40s and 50s are the "middle age" where people would settle down into a routine with their career, raising their kids, etc., and then after that they'd start thinking about retirement and the "golden years" so to speak. If someone has followed this sort of pattern in their own life, it's probably pretty jarring to think of yourself as only just approaching the tail end of your establishing phase of your adult life and then have it pointed out that you might be past the midpoint of your lifespan.
That said, I've also always found the terms "middle aged" and "midlife crisis" to be weird due to them not really making a lot of sense literally. Idioms like this seem pretty common in everyday language though, to the point of being standardized; as a young kid I was shocked when my punishment for some minor thing that was supposed to be for the "weekend" extended into Sunday despite the fact that our family calendar clearly showed it as the beginning of the week, not the end. It's not just language either; the earliest New Year's Eve I can remember, my parents had a very hard time convincing me that we had changed from December 31 to January first at 12:00 because it seemed so unreasonable to me that 12 would be the first hour of the day instead of the last, and I felt cheated that I didn't get to stay up until 1:00 like I expected.