More than likely, the reason others avoid you is because once it became a distinct group huddled around a table nerding out about a technical board game, it became intimidating to join. Hell, I was raised playing D&D and other games, am male, have a computer science degree, and even I feel intimidated walking up to another table of gamers.
The same thing applies in reverse when you're with your mates at a pub. You want to go over and talk to a girl, but she's surrounded by her army of friends who are forming a social barricade that is intimidating to enter.
Those girls don't want to leave their comfortable social circle and risk entering this unknown board game world either. Here are a bunch of guys who all "know" this game and she has to enter into it not just from a social disadvantage, but also a gaming skill disadvantage. That's a tough ask for anyone, male or female. People do not like to enter social situations where they feel isolated.
Even if you were all a bunch of ripped, attractive, friendly, well dressed, wealthy, and obviously social gents, you'd still probably find women don't waltz right up to you and your friends playing a game of Scythe or some other complex strategy game.
If you want people to join your game, you have to go out of your way to make the game environment approachable to other people.
This means playing simpler games, inviting people over, and including other fun things to ease them into it. This means networking outside of just going to a board game cafe and hoping a girl will come through the door. You have to go out and find those wonderful people yourself.
NOTE: this can be really hard, cause the same way you and your friends are hiding out in this board game cafe, so are all those other like-minded individuals. They're also hiding away in basements, back alley cafes, etc, playing board games.
Remember, the reason you're into this hobby is because you've been a gaming fan for years. If we take away all your prior knowledge of games, you too would be anxious to jump right into the thick of it.