31–40 of 235 posts
That's definitely a problem you only get at scale.
https://webcache.googleusercontent.com/search?q=cache:d4DE5B...
This was probably some junior dev's worst first day
Earlier quoted context omitted.
What does that tend to look like? Do you have any examples?
In my experience, it tends to be that you find out one of your components has some tipping point as you scale out horizontally where everything goes to hell. Random example: you scale a service horizontally and suddenly postgres disk space usage goes from "this line looks horizontal" to "we just used 2 years' worth of disk use growth in 40 seconds". And the postmortem is basically like "yep can't really blame anyone…
And the remediation procedure is quite simple too - scale down. If the fleet can't handle it, increase your throttling. If even with increased throttling you can't handle the load and everybody's timing out (but you still can't scale out), you start spinning up new load balancers, and applying weights in DNS where at least some percentage of your callers are getting through while others see full failure.
The key point is in anything related to such an outage, even if it takes a long time to recover, your customers shouldn't see a full outage.
So it was either an entirely different class of problems or Roblex has a huge ops skills gap. I'm betting the former.
edit: never mind, maybe it's the latter after all: https://twitter.com/NIDeveloper/status/1454773313792880640
Earlier quoted context omitted.
My guess is something related to service mesh/discovery. If that goes down, deployment infra also depends on it, and you don't have a solid black start protocol then you're in for a very bad week.
That or DNS. ;-)
Huh, strangely the link doesn't open for me. Seems like an infinite redirect loop. $ curl -i -L https://blog.roblox.com/2021/10/update-on-our-outage/ HTTP/1.1 301 Moved Permanently Server: nginx Content-Type: text/html; charset=UTF-8 Content-Length: 0 X-Redirect-By: WordPress Location: https://blog.roblox.com/2021/10/update-on-our-outage/ X-Powered-By: WP Engine X-Cacheable: bot X-Cache-Group: bot Strict-Transport-Se…
As most of the Roblox community is aware, we recently experienced an extended outage across our platform. We are sorry for the length of time it took us to restore service. A key value at Roblox is “Respect the Community,” and in this case, we apologize for the inconvenience to our community. On Thursday afternoon, October 28th, users began having trouble connecting with our platform. This immediately became our high…
Thank you for posting the text of the blog post. However, I don't understand your TLDR. They clearly were able to identify the root cause, and that is what allowed them to restore service.
> A core system in our infrastructure became overwhelmed, prompted by a subtle bug in our backend service communications while under heavy load.
Translation Something important went down, we couldn't trace the bug.
>This was not due to any peak in external traffic or any particular experience.
Translation We're not blaming Chipotle for this
> most services at Roblox were unable to effectively communicate and deploy.
Translation We couldn't throw more compute at the problem to manage the traffic or to put up a failover copy of anything.
>To the best of our knowledge, there has been no loss of player persistence data
Translation We literally restored our entire infrastructure from scratch and probably our last set of backups. You might have lost some stuff but we don't know what you lost since we couldn't get the last copy of our databases.
Someone on here posted the other day that someone they knew at Roblox said it was their secret store that became overloaded. Presumably a Hashicorp Vault type service (or something similar.) This update appears to support that claim. That's definitely a problem you only get at scale.