There are IMO two angles to this that I identify with (to different degrees):
1 - People wanted Oculus to be a driving force in gaming, and based on recent post-acquisition statement it looks like the company is much more interested in the Metaverse than they are in gaming.
This is certainly a more ambitious goal and a harder problem overall to tackle than gaming, but a lot of Oculus fans don't give two hoots about the Metaverse and just want to fly airplanes, shoot monsters, explore dungeons, or what have you.
There is (IMO a justifiable) fear that the company is going to abandon, or at least scale back, their efforts in promoting Oculus for gaming, in favor of its own goals creating the Metaverse. Like you said, Google Hangouts in VR.
2 - People are excited that the Metaverse a la Snow Crash is actually going to happen, but do not believe Facebook to the be right people to bring this about. The reasoning for this varies, but mostly comes down to privacy protections, prevalence of advertising, and Facebook's seemingly infinite capacity for nickel and diming their users.
In this case people understand perfectly well why VR would be part of Facebook's vision - it just scares the bejeezus out of them.
The perceived bad move for VR in general is that Oculus is going to become a Facebook product, rather than a well-supported, well-evangelized platform on which a whole VR ecosystem can be built (i.e., the original vision). The chief fear is that eventually people are going to by the Oculus purely to access Facebook Presents the Metaverse instead of to access a wide ecosystem of VR products.