This "game" is pure cancer for children, it should never come back.
Get a grip.
Damn, you convinced me.
An Update on Our Outage
91–100 of 235 posts
Re: An Update on Our Outage
#92It’s great to hear that it wasn’t the result of something malicious (e.g. a hack). But this has to be one of the longest outages by a company this big ($50B market cap), at least in the past decade it seems?
What I love most then is that its the largest and it doesn't matter! In this market they will become an even larger company because competency has no correlation to how much money subscribers/advertisers will pay, and none of that has any correlation to what investors will pay! Anybody optimizing for competence is exchanging time for food and shelter and just gets to be reminded of the much harder game they are playi…
If it's down for a couple days every month kids will start playing other games and get hooked on them. Roblox would lose a fair amount of users. Losing users means losing investor money.
Re: An Update on Our Outage
#93Looks like everyone got the cause of this outage wrong on previous speculative posts.
This comment from 2 days ago (the top comment on the post) seems likely accurate to me:
Re: An Update on Our Outage
#94My 3 youngest kids, aged 9, 9, and 8 told me Roblix were having an extended outage, world wide. Good job to the team getting everything back up. Been there, done that. #hugops
> My 3 youngest kids, aged 9, 9, and 8 ... That must have been quite a year in the White household!
Twins, and then probably ~18 months later a younger child.
Re: An Update on Our Outage
#95My guess is Consul shit itself. It works until it doesn't.
For now I prefer Zookeeper because at-least I have experienced most of it's failure modes. i.e they are probably all prone to blowing up but I have lost my eyebrows enough times in ZK explosions that atleast I know what I'm up against.
Consul isn't widely used enough for me to have the same confidence for now, same goes for Vault and Nomad tbh. I really like the design of Vault and it's dynamic secrets system but I am probably just going to implement something similar on k8s secrets so that I don't have to carry around something that might spontaneously combust on me.
Re: An Update on Our Outage
#96Earlier quoted context omitted.
Not far off. Having chatted with friends from the company, consul failed due to a issue with streaming which was introduced in 1.9 ( https://www.hashicorp.com/blog/announcing-hashicorp-consul-1... ) which caused it to have massively reduced tps in certain circumstances. No functional consul meant no vault, no consul/vault meant no nomad, no nomad meant no application servers. Furthermore, the services were down for l…
As someone not familiar with HashiCorp's products, could you give a super high level overview of what consul, vault and nomad are?
Re: An Update on Our Outage
#97This was probably some junior dev's worst first day
Given the scope and the description it's highly unlikely to be caused (or investigated by) juniors. It definitely sounds like the service-discovery/configuration store (which is Consul from what I understand) or their orchestration layer (Nomad) shat itself.
So most likely their most senior SREs/infrastructure folk were the ones sweating bullets.
Re: An Update on Our Outage
#98Earlier quoted context omitted.
Consul is a service mesh. It’s your dynamic service discovery and routing layer. You have systems dynamically allocated in a cluster, they need to reach other systems, you ask consul where they are. Vault is secrets and management. Put secret strings in, get secret strings out (if permitted). Most apps need secrets of some type, and vault is normally discovered via consul. Nomad is an application/workload scheduler.…
> Consul is a service mesh. It’s your dynamic service discovery and routing layer. How does this differ from DNS?
I think service discovery is typically related to e.g., request routing in a way that DNS isn't. DNS could probably replace a lot of the ways service discovery is used today, but it would be a very non-typical setup, which is arguably worse than the custom things people use for it today.
Re: An Update on Our Outage
#99> A key value at Roblox is “Respect the Community,” Yet they still take a 30% cut on all transactions that go to content creators / developers
Isn't that basically the status quo from Apple and Google as well? Not to mention Steam. 30% feels high but at least for Roblox they built the game, I feel like that's much more justified than Google and Apple where they're just collecting a huge chunk because they have a captive market. With Roblox's ecosystem it doesn't feel quite as usurious as Google or Apple doing it on their app stores.
Re: An Update on Our Outage
#100Earlier quoted context omitted.
I mean this is honestly a PR/lawyer blog post. > A core system in our infrastructure became overwhelmed, prompted by a subtle bug in our backend service communications while under heavy load. Translation Something important went down, we couldn't trace the bug. >This was not due to any peak in external traffic or any particular experience. Translation We're not blaming Chipotle for this > most services at Roblox were…
You missed the part where they described the root issue: > Rather the failure was caused by the growth in the number of servers in our datacenters. Their config server ran out of sockets and screwed up everything. Or something like that. Too many servers = nothing works has many ways to unfold and isn't obvious the first time.
>too many servers...
My best advise about scaling, I once read (might have been right here on HN) is many devs know their primary bottleneck. Be it the "master-db", 3rd-party-api, login-service etc. But few devs knows the "2nd slowest/bottleneck" in their service stack. Not saying this was the case here, but always good to think (at least has a mental exercise) about what comes after your primary-bottleneck.
If it's a long standing bottleneck, one day it gets fixed (sw update, better arch) and suddenly everything goes down since now you hitting a new bottleneck you never even thought about scaling.