Not sure I follow the original premise. If Mary knows everything there is to know in physics, then Mary knows about the wave theory of light and the colour spectrum. She may not have experienced the colour red but she is aware of the concept of the colour red. Additionally, while Mary may only receive information in black and white, she perceives in full colour and can see the colours of her own skin and clothes. The…
The point of the thought experiment is to ask if Mary would interpret the experience of seeing the specific color with her own eyes as a new experience. If yes, and given the premise that she knows everything that it is possible to know about the physics of color and the human brain, the conclusion is supposed to be that an experience is not a physical fact of the world. I have written another comment on why I think…
I see what you're getting at and mostly agree (hence the Upvote!). But...
Whether a 1st person qualitative experience is a "physical fact of the world" depends on your level of description. In a sense, it's not an epistemically objective physical fact that I can share with others. However, it is just a brute ontological physical fact of the world that my mind experiences the color red (even if yours doesn't).
This is precisely where we need to clarify "facts of the world". We can't settle this chat until we introduce the notion of subjective vs objective, and epistemic vs ontological.