Earlier quoted context omitted.
The advantage of being a mathematician is that the unifying theories for QM and GR just that, mathematical techniques to unify QM and GR. Unfortunatly, if you are a physicist, I assume your grants at some point will have the words "predictions" or "experiments" in them. But for someone focused on mathematics, string theory, quantum loop gravity (if that is what it's called) and all the others become tools and objects…
In quite a few countries, string theorists (and usually GR people too) are far more likely to be in mathematics departments than in physics. Pretty much for the reasons you describe -- GR was seen in the 60s as an exotic mathematical playground, not quite real physics.
In any case, ideally the framework should touch on geometry eventually as well, but I am not aware of concrete ways to do it. Toposes are considered geometric, but I don't know whether geometric in the sense of GR. The eventual or at least, hopeful, end point is something like [2] where your mathematical framework should ideally in the language of categorical or functorial duality express physical dualities. I don't know how to get to GR, as I say, and should read maybe this [3] but there are many ways to get to QM. One example is through entropy: the generalised abstract set subobjects are partitions and partitions encode entropy through "distinctions", i.e, elements in separate equivalence classes of the partition. The category of sets itself is a model for QM theories via it's inspiration for (symmetric) monoidal categories so perhaps one would want to express in the self-dual language that as well. It's one thing to point to multiple uses of category theory in physics, but it is another to keep track of it in your personal, internal intuition strategy.
[1] https://arxiv.org/abs/1808.01350