Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

An Update on Our Outage

blog.roblox.com

Post reply on HN
Print view View original

121–130 of 235 posts

Re: An Update on Our Outage

#121

Post by sofixa »

jpgvm wrote: post #95

My guess is Consul shit itself. It works until it doesn't. For now I prefer Zookeeper because at-least I have experienced most of it's failure modes. i.e they are probably all prone to blowing up but I have lost my eyebrows enough times in ZK explosions that atleast I know what I'm up against. Consul isn't widely used enough for me to have the same confidence for now, same goes for Vault and Nomad tbh. I really like…

I don't know, anecdotally it seems to me that Consul is more popular than Zookeeper.

But yeah, you should go with the one you know best, especially at any sort of scale.

Vault is the most popular secrets manager out there, and has a lot of advantages over something like Kubernetes secrets ( which aren't even encrypted). Nomad is a bit obscure but IMHO it seems to be gaining momentum.

Re: An Update on Our Outage

#122

Post by jpgvm »

terom wrote: post #101
inside_edition wrote: post #41

Earlier quoted context omitted.

Not far off. Having chatted with friends from the company, consul failed due to a issue with streaming which was introduced in 1.9 ( https://www.hashicorp.com/blog/announcing-hashicorp-consul-1... ) which caused it to have massively reduced tps in certain circumstances. No functional consul meant no vault, no consul/vault meant no nomad, no nomad meant no application servers. Furthermore, the services were down for l…

Choice quotes from their PR piece: https://www.hashicorp.com/case-studies/roblox > We didn’t want to choose any technology that requires the company to drive deep expertise, almost to the point where you have to be a code contributor back into the project to get what you want. Nomad is just very easy to adopt. Better be damn sure you have your 24/7 vendor support contracts in order if and when shit does hit the fan.

To be fair Nomad failed because it's reliant on Consul. This would be the equivalent of having a k8s outage because etcd is down.

Their mistake here is thinking that because you can understand the code of a higher level service that somehow you don't need deep knowledge of it's dependencies.

That is a deeply naive view and they paid the price.

Re: An Update on Our Outage

#123

Post by swiley »

htrp wrote: post #26

As most of the Roblox community is aware, we recently experienced an extended outage across our platform. We are sorry for the length of time it took us to restore service. A key value at Roblox is “Respect the Community,” and in this case, we apologize for the inconvenience to our community. On Thursday afternoon, October 28th, users began having trouble connecting with our platform. This immediately became our high…

This kind of thing will probably become more common as more organizations adopt overly complex cloud deployment automation.

Re: An Update on Our Outage

#124

Post by blowski »

withinboredom wrote: post #108
slg wrote: post #33

My heart goes out to the people who likely had to work crazy hours to fix this, but it really is wild that it was down for so long. What was the last service of this size that was down for 4 days? That is a failure in architecture that goes way beyond whatever the specific cause was here. That post-mortem is going to be a doozy.

This is a benefit of working remote: I work at Automattic and have worked on teams where the members are literally around the world. When shit hits the fan, people are able to hand-off this kind of work to people who are fresh after a good night’s sleep and no one needs (or should need to) work crazy hours.

Automattic always sounds awesome to work for - I know a couple of people who work there as well.

In your experience, when something like this happens, what can leaders do to help in the moment? I'm an engineering manager, and watching people getting over-stressed is much easier to do when you're physically in the office. Working remotely, it's easy to slip in the trap of either assuming no news is good news, or the opposite approach of asking "are you OK?" every 2 minutes. The only solution I've found is to all sit on a call, and pay attention to people's tone of voice, but I don't know whether that actually contributes to people's stress levels.

Re: An Update on Our Outage

#125

Post by jpgvm »

sofixa wrote: post #121
jpgvm wrote: post #95

My guess is Consul shit itself. It works until it doesn't. For now I prefer Zookeeper because at-least I have experienced most of it's failure modes. i.e they are probably all prone to blowing up but I have lost my eyebrows enough times in ZK explosions that atleast I know what I'm up against. Consul isn't widely used enough for me to have the same confidence for now, same goes for Vault and Nomad tbh. I really like…

I don't know, anecdotally it seems to me that Consul is more popular than Zookeeper. But yeah, you should go with the one you know best, especially at any sort of scale. Vault is the most popular secrets manager out there, and has a lot of advantages over something like Kubernetes secrets ( which aren't even encrypted). Nomad is a bit obscure but IMHO it seems to be gaining momentum.

ZK is vastly more popular, it's just much less sexy so people don't write about it. Every production Kafka, Spark, Hadoop, Pulsar, etc cluster is using ZK. Just the Kafka and Hadoop clusters alone probably allows it to rival etcd in footprint.

Consul on the other hand isn't used widely outside of some startups and a few OSS systems like Grafana Cortex/Loki.

When it comes to which one has the most production hours under demanding workloads I think ZK comes out miles ahead of everything, then etcd (because of k8s) then in a very distant third is Consul.

Re: An Update on Our Outage

#126

Post by sofixa »

strictfp wrote: post #107
jpgvm wrote: post #95

My guess is Consul shit itself. It works until it doesn't. For now I prefer Zookeeper because at-least I have experienced most of it's failure modes. i.e they are probably all prone to blowing up but I have lost my eyebrows enough times in ZK explosions that atleast I know what I'm up against. Consul isn't widely used enough for me to have the same confidence for now, same goes for Vault and Nomad tbh. I really like…

My stance is: why even have any of those? Are you running in a local datacenter?

Even on a public cloud, Vault is great for secrets and much better ( has more integrations and is more widely supported) than the cloud vendor's equivalents ( not to mention lock-in).

Nomad is a great orchestrator, with lots of integrations ( e.g. it can just run JARs or firecracker microVMs). IMHO features wise it's better than AWS ECS ( the only cloud orchestrator bar Kubernetes I've used, can't talk about the others), and gives Kubernetes a run for its money on many fronts (native templating, more flexible networking, no YAML, not restricted to containers, etc.).

I wrote about it some time ago, you can take a look if interested:

https://atodorov.me/2021/02/27/why-you-should-take-a-look-at...

Re: An Update on Our Outage

#127

Post by withinboredom »

weird-eye-issue wrote: post #118
withinboredom wrote: post #114

Earlier quoted context omitted.

I assume they have a deal with Apple and Google for less than 30% on in-app transactions. That probably means Roblox is getting between 10-20% of the transaction.

They also have PC and Mac version of it. I have some friends who have played Roblox on PC and my younger cousin plays on iOS. I wonder what the market share looks like between them

That's interesting. I didn't know about the PC version! I'd also be curious of the market share there.

Re: An Update on Our Outage

#128

Post by xorcist »

terom wrote: post #101
inside_edition wrote: post #41

Earlier quoted context omitted.

Not far off. Having chatted with friends from the company, consul failed due to a issue with streaming which was introduced in 1.9 ( https://www.hashicorp.com/blog/announcing-hashicorp-consul-1... ) which caused it to have massively reduced tps in certain circumstances. No functional consul meant no vault, no consul/vault meant no nomad, no nomad meant no application servers. Furthermore, the services were down for l…

Choice quotes from their PR piece: https://www.hashicorp.com/case-studies/roblox > We didn’t want to choose any technology that requires the company to drive deep expertise, almost to the point where you have to be a code contributor back into the project to get what you want. Nomad is just very easy to adopt. Better be damn sure you have your 24/7 vendor support contracts in order if and when shit does hit the fan.

> > We didn’t want to choose any technology that requires the company to drive deep expertise,

That's a beautifully concise quote which neatly summarizes what contemporary IT values.

Re: An Update on Our Outage

#129

Post by joe-collins »

aahortwwy wrote: post #10

> We will publish a post-mortem with more details once we’ve completed our analysis, along with the actions we’ll be taking to avoid such issues in the future. Excellent. That should be a very interesting read. > In addition, we will implement a policy to make our creator community economically whole as a result of this outage. Good to hear.

That latter statement wins a lot of respect in my book. I'm not sure I could name any other platform or game company that I would expect to see go so far.

Re: An Update on Our Outage

#130

Post by sofixa »

jpgvm wrote: post #125
sofixa wrote: post #121

Earlier quoted context omitted.

I don't know, anecdotally it seems to me that Consul is more popular than Zookeeper. But yeah, you should go with the one you know best, especially at any sort of scale. Vault is the most popular secrets manager out there, and has a lot of advantages over something like Kubernetes secrets ( which aren't even encrypted). Nomad is a bit obscure but IMHO it seems to be gaining momentum.

ZK is vastly more popular, it's just much less sexy so people don't write about it. Every production Kafka, Spark, Hadoop, Pulsar, etc cluster is using ZK. Just the Kafka and Hadoop clusters alone probably allows it to rival etcd in footprint. Consul on the other hand isn't used widely outside of some startups and a few OSS systems like Grafana Cortex/Loki. When it comes to which one has the most production hours und…

ZK isn't sexy, we agree. IIRC Kafka no longer requires it? And IIRC it was one of the main downsides of Kafka, that it required a whole ZK cluster.

etcd doesn't compare directly to ZK and Consul, because it's just a distributed KV store. You can build service discovery on top of it with custom tooling, but doesn't come out of the box. Consul does that + service discovery + service mesh + (since last week ) API gateway. On one hand, it's lots of stuff packed together, on the other hand it's useful. I started with it for SD, but when KV was also needed it was easy to add.

I'm not too familiar with ZK, but isn't it also mostly a distributed KV store? Does it have things like Consul's health checks, prepared queries, DNS interface, etc?

Consul is more widely used than that, like Cloudflare and Criteo at huge scale.

Post reply on HN