I don't recall seeing this sort of post-mortem from a gaming provider before. Really cool to see! Kudos, Halo team and Microsoft!
MCC (Master Chief Collection) Server Incident Summary
21–30 of 35 posts
Re: MCC (Master Chief Collection) Server Incident Summary
#22So if I wanted to refer to a group of them, would they be called 'Masters Chief'?
I don't believe so. The full E-9 "title" in the US Navy is Matter Chief Petty Officer, so plural would be Master Chief Petty Officers
Re: MCC (Master Chief Collection) Server Incident Summary
#23I know we normally never hear about this stuff from game compaies at all, but it would be nice to have a little bit more detail. > Finally, it was identified that there had been some updates behind the scenes to the servers we use to relay STUN traffic as part of the ICE process which had resulted in misconfigurations. What updates? How were they tested (or not)?
The thing with using cloud services is that sometimes the hosting provider can make changes to configurations that have impacts (such as monitoring software that steals precious CPU at seemingly random moments, or network topology hardware that impacts connectivity)... without it necessarily being under your own control. I've had a few high-visibility incidents where, after investigation, the buck stopped with us eve…
Of course, that's totally understandable - I recently root caused an incident to "AWS likely changed how this component behaves at some point in a particular 2 month timespan."
But I think there's a lot more to be learned by sharing how STUN changed, what the new behaviour is, what the intent of the change was, how it was tested, etc.
For a counter-example of the level of detail I'd like to see, I saw this [1] DataDog incident report go by on Twitter this morning. This is straight up awesome and more detailed that most of our internal incident reports. I definitely learned a lot from reading it.
1: https://www.datadoghq.com/blog/engineering/grpc-dns-and-load...
Re: MCC (Master Chief Collection) Server Incident Summary
#24The most interesting part of this to me is that they have a path, both in code and in process, to fallback to peer-to-peer if stuff breaks. That's pretty impressive.
Now that big game companies are really starting to shut down old servers en masse it should be the default, really.
https://www.gamespot.com/articles/ubisoft-shuts-down-online-...
Re: MCC (Master Chief Collection) Server Incident Summary
#25Interesting that they call their service UDS. I was under the impression that they used PlayFab.
Halo Master Chief Collection was already released for 4 years (2014) when MSFT bought PlayFab (2018)
Re: MCC (Master Chief Collection) Server Incident Summary
#26The most interesting part of this to me is that they have a path, both in code and in process, to fallback to peer-to-peer if stuff breaks. That's pretty impressive.
Now I'm very curious how the P2P matchmaking is bootstrapped.
Re: MCC (Master Chief Collection) Server Incident Summary
#27Anyone else having issues viewing this on Firefox? I can see the whole webpage for an instant, then everything disappears, then it is "slowing down my browser".
I am but only with uBlock Origin turned on. uBlock Origin blocks something and then the page goes into an infinite loop requesting https://wpcontent.svc.halowaypoint.com/purchase-content/game... and https://wpcontent.svc.halowaypoint.com/content-ratings/esrb/... over and over again. It is also rapidly using up memory so eventually you run low and the whole browser starts performing poorly. Whitelisting the site in uB…
Odd, maybe another extension in combination is causing that issue? I'm running FF 100.0 (wow, I remember FF 3.5) with UBO and Privacy Badger.
Re: MCC (Master Chief Collection) Server Incident Summary
#28I don't recall seeing this sort of post-mortem from a gaming provider before. Really cool to see! Kudos, Halo team and Microsoft!
Roblox did a very interesting and similar write up at the beginning of this year regarding a large outage they experienced at the end of 2021: https://blog.roblox.com/2022/01/roblox-return-to-service-10-...
Re: MCC (Master Chief Collection) Server Incident Summary
#29The most interesting part of this to me is that they have a path, both in code and in process, to fallback to peer-to-peer if stuff breaks. That's pretty impressive.
Also interesting and impressive: Halo multiplayer fans have modded the original Xbox version of Halo to act as a dedicated server for Xbox LAN multiplayer[1] that serves the same architectural role as 343's UHS does for MCC online play.
Re: MCC (Master Chief Collection) Server Incident Summary
#30Anyone else having issues viewing this on Firefox? I can see the whole webpage for an instant, then everything disappears, then it is "slowing down my browser".
I am but only with uBlock Origin turned on. uBlock Origin blocks something and then the page goes into an infinite loop requesting https://wpcontent.svc.halowaypoint.com/purchase-content/game... and https://wpcontent.svc.halowaypoint.com/content-ratings/esrb/... over and over again. It is also rapidly using up memory so eventually you run low and the whole browser starts performing poorly. Whitelisting the site in uB…
I am running FF 99 with only uBlock Origin and I had no slowdowns