I find that playing a game in VR is like listing to music in 5.1: It requires a lot of physical space and setup, which then limits the audience to a dedicated niche.
The limitation, in this case, is being affluent enough to have a large empty indoor space to dedicate to VR so you don't bump into things as you walk around; and the desire to calibrate the thing correctly. It's like setting up a really good sound system to listen to music in 5.1; really fun if you like that kind of thing, but "meh" to most people who are happy with a smart speaker.
In my case, when I played a friend's VR setup, he had to have his entire living room completely empty and the system calibrated correctly. Even though his living room was rather large for a Boston-area apartment, it just wasn't large enough, and I kept bumping into a specific wall where his system was miscalibrated.
Such a system will never work in a typical tiny Tokyo, NYC, or Hong Kong apartment; but I could see it selling being a good niche in suburban areas where it's common to have large basements.