i’m very interested to hear more! it’s hard to speculate without more details, but based off the bits here i’m suspecting that their load balancing/service discovery system hit an issue that resulted in a death spiral. often times a lot of these systems are built with subtle nonlinear scaling bottlenecks that also don’t allow the system to fail into a stable state.
An Update on Our Outage
11–20 of 235 posts
Re: An Update on Our Outage
#12Huh, strangely the link doesn't open for me. Seems like an infinite redirect loop. $ curl -i -L https://blog.roblox.com/2021/10/update-on-our-outage/ HTTP/1.1 301 Moved Permanently Server: nginx Content-Type: text/html; charset=UTF-8 Content-Length: 0 X-Redirect-By: WordPress Location: https://blog.roblox.com/2021/10/update-on-our-outage/ X-Powered-By: WP Engine X-Cacheable: bot X-Cache-Group: bot Strict-Transport-Se…
Same, I can open https://blog.roblox.com/2021/10/ and other linked blog posts, but not this specific one.
Re: An Update on Our Outage
#13Huh, strangely the link doesn't open for me. Seems like an infinite redirect loop. $ curl -i -L https://blog.roblox.com/2021/10/update-on-our-outage/ HTTP/1.1 301 Moved Permanently Server: nginx Content-Type: text/html; charset=UTF-8 Content-Length: 0 X-Redirect-By: WordPress Location: https://blog.roblox.com/2021/10/update-on-our-outage/ X-Powered-By: WP Engine X-Cacheable: bot X-Cache-Group: bot Strict-Transport-Se…
Same with Firefox:
> The page isn’t redirecting properly
> An error occurred during a connection to blog.roblox.com.
Re: An Update on Our Outage
#14There's a redirect loop between https://blog.roblox.com/2021/10/update-recent-service-outage... and https://blog.roblox.com/2021/10/update-on-our-outage/
I'm unable to open the link and seems Wayback Machine cannot archive it either.
Re: An Update on Our Outage
#15Did they run out of internal ipv4 address space somehow? I’ll be curious for the post-mortem, this is definitely the longest outage of a global web service I can remember in… forever? This incident puts them under 2 9’s.
My guess is something related to service mesh/discovery. If that goes down, deployment infra also depends on it, and you don't have a solid black start protocol then you're in for a very bad week.
Re: An Update on Our Outage
#16It’s great to hear that it wasn’t the result of something malicious (e.g. a hack). But this has to be one of the longest outages by a company this big ($50B market cap), at least in the past decade it seems?
Re: An Update on Our Outage
#17Earlier quoted context omitted.
Not a lot of details here but I'd suspect it's yet another example of autoscaling policies gone wrong.
What does that tend to look like? Do you have any examples?
In my experience, it tends to be that you find out one of your components has some tipping point as you scale out horizontally where everything goes to hell. Random example: you scale a service horizontally and suddenly postgres disk space usage goes from "this line looks horizontal" to "we just used 2 years' worth of disk use growth in 40 seconds".
And the postmortem is basically like "yep can't really blame anyone for not knowing this weird ass thing or unfound bug".
Software, man.
Re: An Update on Our Outage
#18This was probably some junior dev's worst first day
Re: An Update on Our Outage
#19Huh, strangely the link doesn't open for me. Seems like an infinite redirect loop. $ curl -i -L https://blog.roblox.com/2021/10/update-on-our-outage/ HTTP/1.1 301 Moved Permanently Server: nginx Content-Type: text/html; charset=UTF-8 Content-Length: 0 X-Redirect-By: WordPress Location: https://blog.roblox.com/2021/10/update-on-our-outage/ X-Powered-By: WP Engine X-Cacheable: bot X-Cache-Group: bot Strict-Transport-Se…
VPN from somewhere else and use a new browser
Most likely your edge node and your client side have cached something not ready yet
Re: An Update on Our Outage
#20Bad timing too, I bet holiday events like Halloween are a big deal in the roblox culture