Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

An Update on Our Outage

blog.roblox.com

Post reply on HN
Print view View original

41–50 of 235 posts

Re: An Update on Our Outage

#41

Post by inside_edition »

lwansbrough wrote: post #31

Someone on here posted the other day that someone they knew at Roblox said it was their secret store that became overloaded. Presumably a Hashicorp Vault type service (or something similar.) This update appears to support that claim. That's definitely a problem you only get at scale.

Not far off. Having chatted with friends from the company, consul failed due to a issue with streaming which was introduced in 1.9 (https://www.hashicorp.com/blog/announcing-hashicorp-consul-1...) which caused it to have massively reduced tps in certain circumstances. No functional consul meant no vault, no consul/vault meant no nomad, no nomad meant no application servers.

Furthermore, the services were down for long enough that the asset caches went fully cold, so spinning back up to 100% capacity would put far more load than usual on the servers meaning recovery needed to be a very slow incremental rollout to warm them.

Messy situation all around, but if you want the real root cause pay attention to the consul change log over the next few weeks I’d say.

Re: An Update on Our Outage

#42

Post by judge2020 »

htrp wrote: post #39
hn_throwaway_99 wrote: post #28

Earlier quoted context omitted.

Thank you for posting the text of the blog post. However, I don't understand your TLDR. They clearly were able to identify the root cause, and that is what allowed them to restore service.

I mean this is honestly a PR/lawyer blog post. > A core system in our infrastructure became overwhelmed, prompted by a subtle bug in our backend service communications while under heavy load. Translation Something important went down, we couldn't trace the bug. >This was not due to any peak in external traffic or any particular experience. Translation We're not blaming Chipotle for this > most services at Roblox were…

> We literally restored our entire infrastructure from scratch and probably our last set of backups. You might have lost some stuff but we don't know what you lost since we couldn't get the last copy of our databases.

What a wild extrapolation. The line in the blog post is not them writing to hint at what might have happened, it's so that the customers reading don't think this means they lost any of their data/items/etc. The Roblox marketplace is vast and high-volume; if anyone lost data from them restoring a backup, we'd already know about it.

Re: An Update on Our Outage

#43

Post by ZephyrBlu »

inside_edition wrote: post #41
lwansbrough wrote: post #31

Someone on here posted the other day that someone they knew at Roblox said it was their secret store that became overloaded. Presumably a Hashicorp Vault type service (or something similar.) This update appears to support that claim. That's definitely a problem you only get at scale.

Not far off. Having chatted with friends from the company, consul failed due to a issue with streaming which was introduced in 1.9 ( https://www.hashicorp.com/blog/announcing-hashicorp-consul-1... ) which caused it to have massively reduced tps in certain circumstances. No functional consul meant no vault, no consul/vault meant no nomad, no nomad meant no application servers. Furthermore, the services were down for l…

As someone not familiar with HashiCorp's products, could you give a super high level overview of what consul, vault and nomad are?

Re: An Update on Our Outage

#44

Post by suresk »

atomicnumber3 wrote: post #17
reilly3000 wrote: post #8

Earlier quoted context omitted.

What does that tend to look like? Do you have any examples?

In my experience, it tends to be that you find out one of your components has some tipping point as you scale out horizontally where everything goes to hell. Random example: you scale a service horizontally and suddenly postgres disk space usage goes from "this line looks horizontal" to "we just used 2 years' worth of disk use growth in 40 seconds". And the postmortem is basically like "yep can't really blame anyone…

I think there are two big things that can be really hard with scalability -

1) There are a lot of dimensions

I think it is fairly common to focus on a throughput-related dimension like requests per second and test that very thoroughly, while not paying enough attention to other aspects. We have services where, if we were told requests per second were going to go up 100x tomorrow, I wouldn't be too concerned, but if we were told request size were going up 2x, I'd be freaking out.

Even when you think you've identified all of the dimensions, there are usually ones you missed and/or they interact in weird and unexpected ways.

2) Performance can appear to be linear when it really isn't

So many outages I've seen have resulted from everything being fine until some threshold is hit, at which point it doesn't start to slowly degrade, but instead immediately explodes. Often times due to feedback loops (GC activity in garbage-collected languages often can behave this way) or because some cache overflowed (data that is queried at a high rate no longer fitting in memory on a DBMS comes to mind).

Re: An Update on Our Outage

#46

Post by [deleted] »

htrp wrote: post #39
hn_throwaway_99 wrote: post #28

Earlier quoted context omitted.

Thank you for posting the text of the blog post. However, I don't understand your TLDR. They clearly were able to identify the root cause, and that is what allowed them to restore service.

I mean this is honestly a PR/lawyer blog post. > A core system in our infrastructure became overwhelmed, prompted by a subtle bug in our backend service communications while under heavy load. Translation Something important went down, we couldn't trace the bug. >This was not due to any peak in external traffic or any particular experience. Translation We're not blaming Chipotle for this > most services at Roblox were…

[deleted]

Re: An Update on Our Outage

#47

Post by kaesar14 »

slg wrote: post #33

My heart goes out to the people who likely had to work crazy hours to fix this, but it really is wild that it was down for so long. What was the last service of this size that was down for 4 days? That is a failure in architecture that goes way beyond whatever the specific cause was here. That post-mortem is going to be a doozy.

The last outage of this kind I can even remember is the summer PSN was down for like, a month?

Re: An Update on Our Outage

#48

Post by gorbypark »

ZephyrBlu wrote: post #43
inside_edition wrote: post #41

Earlier quoted context omitted.

Not far off. Having chatted with friends from the company, consul failed due to a issue with streaming which was introduced in 1.9 ( https://www.hashicorp.com/blog/announcing-hashicorp-consul-1... ) which caused it to have massively reduced tps in certain circumstances. No functional consul meant no vault, no consul/vault meant no nomad, no nomad meant no application servers. Furthermore, the services were down for l…

As someone not familiar with HashiCorp's products, could you give a super high level overview of what consul, vault and nomad are?

Consul is a service mesh, vault is secret storage and nomad is for workload orchestration.

Re: An Update on Our Outage

#49

Post by judge2020 »

ZephyrBlu wrote: post #43
inside_edition wrote: post #41

Earlier quoted context omitted.

Not far off. Having chatted with friends from the company, consul failed due to a issue with streaming which was introduced in 1.9 ( https://www.hashicorp.com/blog/announcing-hashicorp-consul-1... ) which caused it to have massively reduced tps in certain circumstances. No functional consul meant no vault, no consul/vault meant no nomad, no nomad meant no application servers. Furthermore, the services were down for l…

As someone not familiar with HashiCorp's products, could you give a super high level overview of what consul, vault and nomad are?

The home pages for the products are helpful

https://www.consul.io/ https://www.vaultproject.io/ https://www.nomadproject.io/

Re: An Update on Our Outage

#50

Post by djbusby »

cottager2 wrote: post #30
vmception wrote: post #23

Earlier quoted context omitted.

What I love most then is that its the largest and it doesn't matter! In this market they will become an even larger company because competency has no correlation to how much money subscribers/advertisers will pay, and none of that has any correlation to what investors will pay! Anybody optimizing for competence is exchanging time for food and shelter and just gets to be reminded of the much harder game they are playi…

Maybe your definition of competence is wrong? A lot of people can build systems with some number of 9s of availability, but how many people can build a 50B company?

I can do the 9s. So, that's common I'd wager.

$50B is hella hard tho. It's hard work to just get to $10M (w/o outside money) and even with outside money 100M is still very hard.

Post reply on HN