The interesting thing here is that Microsoft (in so far as Microsoft's years of PR will have you believe) agrees with you: the Xbox and "home media center"
is a tiny part of the whole puzzle; it's just a currently somewhat successful part of the puzzle.
Have you ever read Bill Gates' The Road Ahead? The book hasn't aged well, of course. (To put its age in perspective, it was a big deal that it launched with an accompanying Multimedia CD-ROM.) But a large gist of the book is exactly that point that technology will (and should) drift into the foundations of the home, seamlessly touching nearly every device and surface in the home. Most of the details are obsolete by now, but I do think the overall vision is still inspiring. (Even to this day, Microsoft keeps and updates a "smart home" concept for PR usage that they can play act what a seamlessly integrated supere home may be.)
Things are moving slowly because they have to. It's been 20 years since The Road Ahead was published (!). There's a sense that if Microsoft had found some magic way to jump straight to whole home automation and seamless device linking in those 20 years they would have jumped on it.
The interesting thing is that even in the world we have you can see that DNA threaded through their efforts. The Xbox is the TV set top box that succeeded after multiple, multiple attempts at Microsoft before they found that the intersection of Games and Media Center was the way to get into that space.
The 360 tried to extend your media center from a home media server you probably didn't own. The Xbox One tries to extend itself, your Xbox and its access to your TV, to every Windows 10 device you probably do own... Yes, Xboxen have to standalone as game playing warriors on the threshold of your television, but you can see Microsoft trying, however successful you think they may be, to expand that deeper into your house...
Then there's other "standalone" pieces of the puzzle that have been R&D efforts and direct learning experiences. Kinect put a UX R&D lab in a bunch of people's living rooms. There's no HoloLens without Kinect, and HoloLens too has that DNA of trying to get to a place where everything in your space (not just house, but maybe even your office) is seamlessly digital. There's Cortana very much trying to be your day-to-day Jarvis, at least for appointment scheduling and reminders today.
The reason I ramble down this tangent with you, is that the interesting thing today, that I don't think you appreciate given your opening paragraph, is the difference between what Microsoft wants these days, versus what Apple and Google are trying. Some of it comes from the maturity of thinking about all of this stuff for more than two decades already (and building many, many baby steps toward it) combined with fresh executive leadership. Microsoft has backed away again from trying to build "the Microsoft Home" and is content to once again be in a place similar to where the company started (and those bits of DNA that remain from The Road Ahead). Microsoft wants you to have an awesome home, regardless of which devices you choose to use, but they want those devices to be full of Microsoft software and connected to Microsoft's cloud. They don't mind being exclusive again, they seem to want to be the company you come to for the best apps and the best tools to develop your own apps.
The Xbox One wants to extend itself to every Windows 10 device that you already own, as well as Android and iOS. The Xbox app is installed by default for home Windows 10 users to make it easy; it's just an app store download away on the other platforms. Cortana wants to extend "herself" to every Windows 10 device that you already own, as well as Android and iOS. Again, Cortana is there by default in every Windows 10 install, but available for Android and iOS, too.
I cannot tell you if this New Microsoft will be any more successful at making our homes amazing seamlessly integrated, beautiful systems, than the Old Microsoft, but I'm willing to bet that they might be, and I'm willing to hope that it will be faster than another 20 years away as so many of the puzzle pieces start to come together (Xbox integration with Cortana; Better cross-device Cortana "hand off"; "Cloud Notifications" merging all of your device notifications and dismissal; ...), and maybe much, much faster than your pessimistic 30-50 year horizon...
«So far, we've got nothing worthy of being called amazing.»
Per Gibson's Law, "The future is already here, it is unevenly distributed" is the corollary that "The things we have today seem ordinary and boring, as we forget that they are really the future, distributed to us today". There are technologies we take for granted today that make the "big amazing future" things proposed and explored in The Road Ahead seem quaint by comparison. Sometimes we forget to stop and appreciate how amazingly the bits of future that we have that are already here.