I mention reproductive role (gametes etc.) because that’s what sex is, not because it’s the sole feature of interest. Why we care about someone’s sex is a different matter. Sometimes we don’t! You can sell someone a coffee, and their sex has little bearing on the situation. But sometimes we do care about their sex, because their sex has material consequences. Perhaps sports are the most relevant example to this discussion about passports. The IOC uses “passport gender” to assess eligibility, which is possibly why the current administration is insisting that passports reflect sex, not gender, because sex is what matters to sporting competition. Gender is far less relevant.
There are many other situations where sex and its material consequences matter more than gender. Yes, it’s not gametes that make a man run faster, but there’s a clear causal chain between gametes, sexual differentiation pathways, gonads, and testosterone.
Incidentally, it’s a really weird idea that we can’t tell someone’s sex without looking in their pants, or without some advanced scientific analysis. In reality, outward appearance is extremely highly correlated with sex, and humans (and a lot of animals!) have evolved to be experts at detecting sex, sometimes at a great distance. Visual appearance, gait, voice, behavior, and more, are enough clues to not just make a guess at sex, but to do so with very high accuracy. Probably upwards of 99%. Official documentation is there to clarify the 1%.