Sure! I think I understand what's going on here. I think we're having a few different misunderstandings, let me see if I can describe them.
By "human sex" I'm referring to everything that developmentally contributes to a human sexual genotype or phenotype. Here is a fun textbook example of all the things that go into human sexual development: https://open.lib.umn.edu/evosex/chapter/8-7-variations-in-hu...
From all these variables alone, we know there is some kind of spectrum to human sexual development. When we say something like "binary" that specifically means a discrete data type: true or false (nothing in between). Binary explicitly means there are two options and nothing else (discrete). For example, humans and butterflies, there's zero half-human half-butterfly hybrid. They're discrete. Most things in real life, however, are some form of continuous spectrum, and that's where "spectrum" comes is. We already know based on how many variables go into human sexuality that it's some form of spectrum for an individual, at the very, very least: male, some kind of intersex, or female. It's not binary (two options) for individuals.
(NOTE: also this is just for humans. As I mentioned, "female" and "male" aren't even that useful in a large part of life on Earth. There's a type of sea slug that essentially has penis battles to determine who donates which gamete, essentially, since every individual is capable of both at all times. As I mentioned before, all clownfish are hermaphrodites, they can switch if they need to. Many frogs and lizards can be too. That's why "female" and "male" are such abstract concepts, trying to describe generalized reproduction can be pretty gnarly once you get into any kind of detail, so no wonder the terms are fairly overloaded).
As I mentioned before, some of the factors that go into human sexual development include (1) number of chromosomes, (2) number of X's and Y's, (3) SRY gene on the Y chromosome, if present, (4) human sex hormones including the many forms of estrogen, testosterone, androgens etc, (for this discussion, probably not hormones like FSH, LH etc), (5) the receptors for these hormones. All of these things (and more) go into human sexual development. But you might have already noticed an issue with it: for example, all humans, regardless of sexual phenotype generally have both testosterone and androgens ("male" hormones) and various forms of estrogen ("female" hormones), so how does that work? How can we "measure" "sex"? What even is "sex"?
There's a few different abstractions at play. There's general abstractions like "female" and "male," which have multiple meanings. When we're making big generalizations about reproduction, it's helpful to talk about two general types of roles for mammals. For example, the male and female gametes for mammals are ova and sperm. We can also use it to talk about male and female hormones, estrogen, androgens, progesterone, FSH, LH, testosterone, to name a few. These are useful for talking about general reproductive abstractions, but each individual has aspects from both of these abstractions (estrogen, testosterone, androgens), so it's not a one-to-one mapping. You can't say "oh, human sexuality is determined by chromosomes," because it's not: XXY, XYY, and SRY gene all exist. You can't say "human sexuality is completely based on gametes" because hermaphrodites can have both ova and sperm, so are they male or female? That kind of thing. There's no "one determining" factor for human biological sex. Multiple things go into it, and therefore it's some kind of spectrum based on all the factors that go in.
When we talk about gender this becomes even more readily apparent. There's no "one determining factor" for what makes a woman or a man. It can't be chromosomes, SRY, hormones, gametes, or any one thing alone. We also know that it can't purely be about reproduction: infertile or sterile men and women can still be considered men and women. And this is just English, there are plenty of other languages that have had and have always had more than two main gender roles etc. So what on earth is it? The truth is, it's literally just an arbitrary line in the sand that we're trying to come to some form of consensus on. In general, we've found the most respectful way to do this, is to treat everyone as fully functional humans and have them self-report based on their language, culture, experience et cetera. Language and words are constantly changing and updating with our understanding, so whatever we decide on today, might change in the future too and that doesn't matter either.
Does that help the discussion at all? This is all fairly standard, there are quite a few textbooks on it, including the one I linked if you're interested.