I think the differences are far less fundamental than you think.
Some places indeed do have unisex toilets nowadays, and while I think few if any people consider separate toilets explicitly offensive, at least so far, they do actually pose a problem for some people, which is why I think it is not unreasonable to consider switching to unisex toilets everywhere. The problem with the current scheme of toilet separation is that there are both people who do not identify as either explicitly male or female, and in particular people who identify as a different gender than their biological one, for those people the scheme doesn't really work (and actually sometimes leads to real-world conflicts when they are perceived by others to be overstepping boundaries when they go to the "wrong" toilet). And if you think about it, the separation doesn't really make all that much sense. After all, you don't have separate toilets at home either, do you? And it's not like you couldn't have any privacy in public toilets nowadays, which a gender separation might be a crude approximation for.
Also, you have to distinguish between enforced categorisations and self-identification. A "ladies night out" probably is not really strictly about genitals, but rather a heuristic description of what kind of activities are planned, so if you happen to be a biologically male transgender, who identifies as female, and who is interested in taking part in that activity, behaving in that social role, you hopefully would not be excluded due to your biology. In particular, there is a difference between (biological) sex and social roles traditionally associated with certain biological sexes, commonly called the gender. This whole fight is not about getting rid of those roles, but rather about the freedom for each individual to choose their role independently of their biology (and also the freedom to reject both roles). Just as you can like or dislike swimming, you can like or dislike behaving according to what is traditionally considered "male" or "female" behaviour, and you should be able to choose to behave according to any of those roles as you like to, and not be forced by society to fill one of the roles because of your biology, just as you can freely choose to pursue swimming, or to not pursue swimming. That does not mean that we should ban swimming or "behaving female" or "behaving male".
Also, all of this obviously is not to be confused with a distinction that is based directly on biologically determined differences. If you do only have one kind of genitals, then services for the other kind of genitals probably aren't for you, just as pedicure is not for people who don't have legs. But that's not exactly something that needs regulation or social pressure to enforce.
And if you look at it from that perspective, it might actually be more analogous to race than you think: Yes, inviting only white friends would be considered weird. But inviting people "for a day of doing what was traditionally what white people did" (and by that I don't mean anything "anti-black" or anything, just typical cultural differences)? You probably would not phrase it that way, but conceptually I don't think anyone would find anything wrong with that, and I guess part of why you would not phrase it that way is simply that it's the "majority/privileged/normal" thing anyhow, so it's unusual to name it explicitly, but looking at it from the other side, consider, for example, "black music": It would be perfectly fine to have a "black music festival", but inviting only black musicians or only admitting black audience would be racist, and there also is no stigma anymore for white people to be interested in "black music", listening to it and even participating in it. It's a culture that was traditionally associated with "race", and we dropped the exclusion, but we still kept the culture, and even name it according to where it came from.
As such, I would say: No, gender is different and equal, but separation as it is commonly understood should not have any place there. Just as different kinds of sports are not "separate but equal". You should be free to engage in any sports activity that you like, and you should be free to engage in any gender role that you like, and just as you usually won't find many swimmers swimming in the football stadium, you won't find many people into male roles at places where people meet "to be female", but it's noone's business to enforce some separation.