When an artist draws with a display-less graphics tablet, the hand is in one place, but they watch and use an abstraction of it on a display, displaced in space, orientation, and scale. It can take some getting used to. Touchpads and cursors have similar non-trivial relationships. And similarly, there's no reason for your hands to be positioned at the end of your arms (as with "throwy hands"). Nor, with non-novice users, need any other physical correlation be maintained if it's useful to ignore it.
When you use office apps, there's little attempt to maintain physical-world properties like say object permanence. Delete a word, and it "goes someplace, because it can't just vanish"? That would be silly. There's no need to emulate physical paper, or pens, or rubber erasers, or cut-and-paste with an xacto knife, paper fragments, and gooey paste from a jar. It can sometimes be useful or artistically fun to do so, but that's a UX choice.
Or say you're looking at a whole-desktop "window", placed uncomfortably close and large, to be readable despite current HMD's limited angular resolution. When you turn your head, how much does it move? With what velocity profile? And with what alterations in appearance? Skeuomorphism suggests it appear as a real physical object, nailed in space. I found I preferred a quantized 2-3x displacement, to reduce head motion, and enabled it by transiently increased transparency (including "comfort mode" clipping from fov), with slightly delayed motion and added jiggle, so other visual cues could dominate wrt balance. And the shape distorted to maintain pixel alignment, because given my weak gpu, alignment was worth more than temporal antialiasing when reading small fonts. YMMV, as just what combinations of cues are fine vs intolerable varies so much among people.
The common thread is aphysicallity. Skeuomorphism can be useful when onboarding novices, but is later often largely abandoned. As with phones. With the current focus on immersive games and novices, that second part tends to be neglected. I joke that while I use XR hardware, I'm not doing XR, because being focused on non-game non-novice UX, and using a custom non-gaming stack, I've no allegiance to the "R", to Reality. Resemblance to reality as UX design smell.