I presume that you also refuse to do talks or workshops where all the speakers, panelists or mentors are of one gender or race? If so, good on you.
Look, you sound like a nice guy, so I don't mean this as a personal attack, but your take on this is a classic example of how good people can fall into the trap of their own privilege. You don't do events where men/boys are not allowed to attend (e.g. girls who code) and that sounds noble and ideal. However, it somewhat ignores the fact that the reason these events have come into existence is that our profession has for decades suffered from massive amounts of systemic discrimination (sexism in particular). Just because it's not encoded in a written set of rules, doesn't mean it hasn't always been present. For men/boys, there are infinitely more events to attend where the gender balance will always be swung widely in their favor.
When a woman developer joins/joined an all-male development team, do you think they have the luxury of being able to say "No, I won't do that"? Sure, there's no rule saying the team should be only male, but that's the "systemic" part. This is the "privilege" I'm referring to. You can say "no", because there are plenty of other talks and workshops for you to take part in. There is little to no negative impact on you from refusing those invites. What if women-only workshops were the predominant event out there?
I don't think events like Girls Who Code are the final solution. I agree with you that, in the end, an industry and events open to everyone is the solution. However, we've reached such a massive imbalance by operating via the status quo that I think things like Girls Who Code, despite being "discriminatory" by a textbook definition, have a net positive effect, rather than the opposite. I'll happily volunteer my time to teach more girls to again when I have the chance. I don't think the lip-service of saying our industry and events are open to both men and women will tip the scales towards equality.