Given that the error resulted from the STUN and ICE servers, which from my understanding exist solely to play a part in the NAT punching process, would this entire situation have been mitigated if things were end-to-end IPV6?
MCC (Master Chief Collection) Server Incident Summary
11–20 of 35 posts
Re: MCC (Master Chief Collection) Server Incident Summary
#12Wow, that was a fun read! I don't envy the people who had to stare at wireshark logs for 3 days though. Oof.
Re: MCC (Master Chief Collection) Server Incident Summary
#13Anyone else having issues viewing this on Firefox? I can see the whole webpage for an instant, then everything disappears, then it is "slowing down my browser".
I am but only with uBlock Origin turned on. uBlock Origin blocks something and then the page goes into an infinite loop requesting https://wpcontent.svc.halowaypoint.com/purchase-content/game... and https://wpcontent.svc.halowaypoint.com/content-ratings/esrb/... over and over again. It is also rapidly using up memory so eventually you run low and the whole browser starts performing poorly. Whitelisting the site in uBlock Origin is a workaround.
Re: MCC (Master Chief Collection) Server Incident Summary
#14Really cool to see this level of transparency on an issue from a multiplayer game dev. Really cool write up
Both Bungie and 343 have done an admirable job (well, compared to other devs) about explaining their network infrastructure etc. Back in the day they did a big talk about how their matchmaking in Halo2/3 worked that I think to this day is still one of the best methods of learning when you're not in the industry yet. I can't recall what it was called though: might be the "Chris Butcher - Recreating the LAN Party Onlin…
And similar to my sibling comment, mostly for anyone else browsing through - the comments on this recent HN post have a lot of resources about game dev stuff in general, which might be of interest (though I also cannot promise directly related X) ): https://news.ycombinator.com/item?id=31084779
Re: MCC (Master Chief Collection) Server Incident Summary
#15Earlier quoted context omitted.
Both Bungie and 343 have done an admirable job (well, compared to other devs) about explaining their network infrastructure etc. Back in the day they did a big talk about how their matchmaking in Halo2/3 worked that I think to this day is still one of the best methods of learning when you're not in the industry yet. I can't recall what it was called though: might be the "Chris Butcher - Recreating the LAN Party Onlin…
Only semi-related, but there was a recent excellent, short podcast interview series with one of Halo 2's multiplayer designers: https://smarturl.it/H2Pod
Halo 2 was my favorite experience of FPS multiplayer in my life so thank you for sharing!
Re: MCC (Master Chief Collection) Server Incident Summary
#16Really cool to see this level of transparency on an issue from a multiplayer game dev. Really cool write up
Both Bungie and 343 have done an admirable job (well, compared to other devs) about explaining their network infrastructure etc. Back in the day they did a big talk about how their matchmaking in Halo2/3 worked that I think to this day is still one of the best methods of learning when you're not in the industry yet. I can't recall what it was called though: might be the "Chris Butcher - Recreating the LAN Party Onlin…
Along these lines is the venerable TRIBES Engine Networking Model whitepaper. It was so good, it was shipped as part of the XDK for a decade. I believe Bungie even leaned on it quite a bit when creating their networking stack.
https://www.gamedevs.org/uploads/tribes-networking-model.pdf
Disclaimer: I work at Microsoft Game Studios, but this comment reflects my own opinions.
Re: MCC (Master Chief Collection) Server Incident Summary
#17I know we normally never hear about this stuff from game compaies at all, but it would be nice to have a little bit more detail. > Finally, it was identified that there had been some updates behind the scenes to the servers we use to relay STUN traffic as part of the ICE process which had resulted in misconfigurations. What updates? How were they tested (or not)?
The thing with using cloud services is that sometimes the hosting provider can make changes to configurations that have impacts (such as monitoring software that steals precious CPU at seemingly random moments, or network topology hardware that impacts connectivity)... without it necessarily being under your own control. I've had a few high-visibility incidents where, after investigation, the buck stopped with us even though technically the studio could have shifted blame to an unanticipated change from the hosting provider.
Disclaimer: I currently work at Microsoft Game Studios, though this comment reflects my own opinions based on experiences unrelated to this incident.
Re: MCC (Master Chief Collection) Server Incident Summary
#18Given that the error resulted from the STUN and ICE servers, which from my understanding exist solely to play a part in the NAT punching process, would this entire situation have been mitigated if things were end-to-end IPV6?
Only in theory. In practice even when IPv6 is in use, people have stateful firewalls that will drop unsolicited connection attempts.
Compared to ipv4 where there is UPnP and NAT-PMP with widespread support in routers, there are protocols to allow clients to reconfigure the router with ephemeral firewall rules, but they are not wide-spread and support is very spotty.
So in practice, users with just IPv6 would have the exact same problems and would be even more likely to depend on STUN and ICE because their firewalls likely won’t support client-side hole-punching correctly
Re: MCC (Master Chief Collection) Server Incident Summary
#19So if I wanted to refer to a group of them, would they be called 'Masters Chief'?
Re: MCC (Master Chief Collection) Server Incident Summary
#20Anyone else having issues viewing this on Firefox? I can see the whole webpage for an instant, then everything disappears, then it is "slowing down my browser".