There are good reasons for male-only organizations as well, mainly to do with having something private of their own that men can bring back to their marriages and homes. It provides a social support network for men (literal fraternity), and by extension, their families, where men are not left dependent on their wives for managing their social lives, as is common in many marriages. The lodges still provide a fabric for civil society that persists across party lines and through periods of strife. To dismiss the lodges as merely sexist is more trivializing than anything else, as we're equal, not indifferent or interchangeable, etc.
Racism doesn't really figure into regular Freemasonry either, as it was founded as a universalist enlightenment society. When you look at who has historically made a point of suppressing them, it's worth asking who the good guys really were. They're a reliable canary for civil liberty. Prince Hall masonry, founded by and for African American men is still going strong in the US, and it is in concordance with most north american grand lodges. The main thing is you can't be an atheist in regular masonry, which is analogous to having a policy against racism in that atheism is traditionally also viewed as an obtuse and unnecessarily irreconcilable belief that can call someones moral judgment into question. The continental/european stuff is different, and much of it isn't connected to the mainstream anglosphere masonic culture.