Earlier quoted context omitted.
But at this point I have to ask, how do you define identity? I'm pretty sure that I could use the strategy I outlined above to separate our objects into three groups - As, Ds and the rest. So how do you define that Bs are not Cs, if there is no possible way for telling the difference?
I'd define identity as the smallest relation holding between all things and themselves. If you want, you can redefine identity in terms of some notion of indistinguishability, but then you'll end up with the odd consequence that identity is not transitive. In other words, you'd have to say that if A is identical to B, B is identical to C, and C is identical to D, it doesn't necessarily follow that A is identical to D…
So at this point I'm not sure if your example is, or is not an issue for a working definition of identity. To circle back to p-zombies, as far as I understand, they are not supposed to be distinguishable from non-p-zombies by any possible means, which includes testing everything against everything.
What if I define the identity test I(a,b) in this way: I(a,b) ↔ ∀i : i(a,b), where i(a,b) is an "indistinguishable" test? This should establish a useful definition of identity that works according to my scenario, and also your last example unless you limit the domain of X and Y to integers from 1 to 4. But in this last case there's absolutely no way to tell there's a difference between 2 and 3, so they may as well be just considered as one thing.
As I said, I need to think this through a bit more, but what my intuition is telling me right now is that the very point of having a thing called "identity" is to use it to distinguish between things - if two things are identical under any possible test, there's no point in not thinking about them as one thing.