Agreed, and I think people adopt this reductive view because it can be quite difficult to reason about objectively. In terms of a framework to channel one's thinking on this, I found this paper useful in understanding the rationale behind defining distinct categories of competitors in sports:
https://www.researchgate.net/profile/Jim-Parry/publication/3...The key takeaway in my view is that the authors make a distinction between "category advantage", which is a systematic, structural, group-based difference that exists before competition even begins, and "competition advantage", which we see play out in competitive events and is based on a mix of factors including skill, preparation, and both innate and trained talent.
Where exactly to draw the line can be somewhat subjective (e.g. in weight classes) but it helps to explain why we have a separate female category: male physiology confers such a significant category advantage that, in open competition, it would limit the ability of female athletes to compete meaningfully and demonstrate their abilities. Having a separate category fulfils this desirable outcome of showcasing and celebrating female athletic excellence.
Often we see calls to add various classes of males, particularly ones who have chosen to identify as women, framed as "inclusion" but from the perspective of who this category is actually intended for it's the opposite. Drawing a clear eligibility boundary around the female category maximises inclusion of female athletes who would otherwise be disadvantaged and excluded.