Sometimes your spade is broken enough that it's hard to use as a spade, or so bent from use that it'd be more reasonable to call it a pickaxe
The social meaning of a "man" or "woman" encompasses a large swath of things. Generally people's mental models of those concepts includes both physical and mental ideas. The issue is, this is not very constant. Some people will inevitably fail to possess some characteristics (like lacking large muscles, or having small breasts). We typically allow these exceptions without questioning it too much. Genitalia, of course, is the exception. In my experience, it is typically regarded as the fundamental cornerstone of gender.
The question is, is that really accurate?
We assign labels to make sense of the world. So, we tend to stick things in the categories where they fit the most (important) attributes. After all, we don't measure the genetics before calling something a dog, we typically see it bark, wag it's tail, play fetch and decide it's probably a dog, even if we haven't seen it before.
If someone looks like a girl, acts like a girl, has had genital recontruction surgery, has a fashion sense, wears makeup, and generally is just another regular person on the street, then for any social interaction they are just like any girl. There is one situation where this isn't true - if they are sexually involved with someone who has the intent of having children. For that purpose, they, like any infertile woman, can't provide children. I can sympathise with a hatred for people trying to force falsehoods on me - I personally have been the annoying atheist friend quite a lot. However, to take that birth genetalia are a cornerstone of gender uncritically doesn't seem very useful, since then the labels "man" or "woman" tell you nothing about how they might act - you may meet a woman, who is perfectly ordinary in every way, who then reveals that she is trans. If birth genitalia is to be the cornerstone of gender, you now have to include someone who looks and acts as any woman would in your definition of "man". To me this seems like it makes the term meaningless, and it makes far more sense to just classify that person as a "woman". Genetalia itself isn't really important to social relationships outside of sex, so for situations outside of sex classifying by it also seems somewhat useless.
The other, much more (imo) defensible argument you expressed was relating socialization. In my experience it is true that socially, boys are conditioned to be more aggressive while girls are often punished for being aggressive. This definitely leads to a difference in how far they are willing to push to fulfill their own needs.
However, assertiveness, to an extent, is a useful trait for anyone to have. Imo it shouldn't be seen as a "masculine" trait, but rather a trait men tend to have, since being able to clearly communicate needs and boundaries, and push back when those needs and boundaries get ignored, is an essential skill for anyone, but one that women might be less practiced in due to often receiving higher backlash for doing so.
Of course, there are non trans women who possess that capability. However, in your experience dealing with trans women, has that made your experience in interacting with them closer to an angry man, or closer to any other aggressive woman? It probably varies case by case. From what I've heard the looks of a woman come with the social pressures of a woman, so I'd wager that an aggressive trans woman, on average, wouldn't differ much from an aggressive non trans woman, simply because, so long as they are perceived as women, their anger is taken extremely seriously on several issues (sexual crimes and harassment) and somewhat less seriously on everything else. My personal theory is that this is due to the assumed threat of physical violence with men, while women are often seen as being incapable of/unlikely to commit violence, leading to two lopsided power dynamics rather than one even one.
Regardless, I think that basing our definition of gender off of genitalia or genetics doesn't provide a super useful mental model for socialization, which I believe to be the primary good use of gender as a concept.
Sports are one area in which I think you're wrong. If growing up on testosterone were a major contributor to advantages, we'd expect to see a lot more trans athletes at the top of the women's league, the way you anecdotally noted we do with buisness. They have been allowed to compete for quite some time now, so the fact that only recently have the media made a circus about a trans athlete in the olympics suggests to me that they likely do not possess any significant advantage - in my personal experience, I saw someone who could previously do 20 pushups when completely unfit struggle to do 1 a year after starting hormones, so I find it kind of hard to believe they retain any significant advantage, given enough time
We can say that we are humans, or evolved monkeys, or extremely processed stardust; all can be correct, but laws only apply to humans, and no one gets out of jail by claiming to be an inanimate object.