Technology is not the problem. Flashback to 1995. I'm a 17 year old PFY invited to beta test a VR headset from a local company. My Uncle was an investor and I was technically savvy and the target market. I had a copy of Descent that supported VR, and while it was a bit of a hassle to get setup, I got it working. The experience was incredible. After the first use I said to myself "this is the future". I had permission…
The CAVE approach projects images onto wall-sized screens (eg, via back-projection), and can be 1-wall up to 5- or 6- walls, if the floor/ceiling is categorized as a "wall".
This is massively expensive compared to the VR that's widespread/affordable these days.
But it's also far more social, where multiple people can be present, though at least in the 1990s the head-tracker only followed one person. And they could easily see each other.
Handwaving, with the right synchronization on the glasses and high frame rates, I could imagine multiple head-trackers. Back then our frame rate was limited by the decay rate of the green phosphor in the projector. These days I suppose a wall of large displays, edge-to-edge, with very high frame rates interleaved for multiple users, might work. Again, I haven't followed what's going on in VR, but it might be a technology which addresses at least some of what you correctly point out.
Carolina Cruz-Neira, the developer of the CAVE (and at the time with the email address "cavewoman"), has some lectures on the social VR and her research, on YouTube. Don't remember which ones specifically talk about it, and couldn't find one in a quick look.