Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

An Update on Our Outage

blog.roblox.com

Post reply on HN
Print view View original

141–150 of 235 posts

Re: An Update on Our Outage

#141

Post by that_guy_iain »

dilyevsky wrote: post #57
slg wrote: post #33

My heart goes out to the people who likely had to work crazy hours to fix this, but it really is wild that it was down for so long. What was the last service of this size that was down for 4 days? That is a failure in architecture that goes way beyond whatever the specific cause was here. That post-mortem is going to be a doozy.

Not necessarily directed at roblox but honestly I’m surprised this doesn’t happen more often given how many teams I see run software they don't understand or don’t even have access to its source code. Edit: but yeah must’ve been tough 72+hrs i hope their version of reliability team can use this to bash the support they need out of the management and not get scapegoated instead

At my company there was a service outage that lasted 2-3 weeks for a specific feature we have. This was caused by everyone quitting and no one having any experience with this service. The rest of the application remained working so it wasn't so noticable to the outside world. But internally and for customers it was massive since it was the billing system that went down.

There was another incident that took down everything that was caused by a Spring cloud config being changed that no one in the company had access to.

Re: An Update on Our Outage

#142

Post by mupuff1234 »

blowski wrote: post #124
withinboredom wrote: post #108

Earlier quoted context omitted.

This is a benefit of working remote: I work at Automattic and have worked on teams where the members are literally around the world. When shit hits the fan, people are able to hand-off this kind of work to people who are fresh after a good night’s sleep and no one needs (or should need to) work crazy hours.

Automattic always sounds awesome to work for - I know a couple of people who work there as well. In your experience, when something like this happens, what can leaders do to help in the moment? I'm an engineering manager, and watching people getting over-stressed is much easier to do when you're physically in the office. Working remotely, it's easy to slip in the trap of either assuming no news is good news, or the o…

The honest truth - if you're an engineering manager and you're no longer anywhere near the code then just at the heat of the moment there really isn't much you can do, aside from what you've been doing: listening on the call and doing a minimal check-in.

The main thing you can do is after all is over to make people feel awesome, give them recognition in all the ways possible: bonus, extra vacation days to compensate for the crunch time, etc

But if you still have your technical chops - jump in the trenches and debug!

Re: An Update on Our Outage

#143

Post by nonameiguess »

slg wrote: post #33

My heart goes out to the people who likely had to work crazy hours to fix this, but it really is wild that it was down for so long. What was the last service of this size that was down for 4 days? That is a failure in architecture that goes way beyond whatever the specific cause was here. That post-mortem is going to be a doozy.

Not a software outage, but the Texas ice storm of February resulted in me not receiving mail for two weeks, not being able to use the streets for at least a week (they never plowed and it just eventually melted), rolling blackout where I had power for about four hours a day, and no Internet service at all for the first two days. No water for most of the first day.

I'd argue keeping basic city services and roads operational is more important than a gaming platform, but city and state governments in the US often seem to not agree and are more than happy to save money skimping on maintenance and resilience.

Re: An Update on Our Outage

#144

Post by tyingq »

inside_edition wrote: post #41
lwansbrough wrote: post #31

Someone on here posted the other day that someone they knew at Roblox said it was their secret store that became overloaded. Presumably a Hashicorp Vault type service (or something similar.) This update appears to support that claim. That's definitely a problem you only get at scale.

Not far off. Having chatted with friends from the company, consul failed due to a issue with streaming which was introduced in 1.9 ( https://www.hashicorp.com/blog/announcing-hashicorp-consul-1... ) which caused it to have massively reduced tps in certain circumstances. No functional consul meant no vault, no consul/vault meant no nomad, no nomad meant no application servers. Furthermore, the services were down for l…

>No functional consul meant no vault

Vault does support other backends, like postgres, etcd, zookeeper, and others. Though if Consul is the backend, Vault is also registered as a service within the Consul mesh.

Re: An Update on Our Outage

#145

Post by strictfp »

jpgvm wrote: post #112
strictfp wrote: post #107

Earlier quoted context omitted.

My stance is: why even have any of those? Are you running in a local datacenter?

Generally you need them even when running in a "Cloud" of some description. Namely because they provide distributed locking primitives at a speed that can't be matched by any other mechanism. If you only need service discovery you can probably get away with whatever you platform provides (EC2 API, k8s API, hell DNS works, etc), similarly if you only need slow master election there are alternatives there too (DynamoDB…

Having built many distributed applications I would hesitate to use ZK/consul directly, mostly since your cloud provider is already providing most type of primitives directly or indirectly in the services they provide.

Re: An Update on Our Outage

#146

Post by nonameiguess »

colordrops wrote: post #84
inside_edition wrote: post #55

Earlier quoted context omitted.

Consul is a service mesh. It’s your dynamic service discovery and routing layer. You have systems dynamically allocated in a cluster, they need to reach other systems, you ask consul where they are. Vault is secrets and management. Put secret strings in, get secret strings out (if permitted). Most apps need secrets of some type, and vault is normally discovered via consul. Nomad is an application/workload scheduler.…

> Consul is a service mesh. It’s your dynamic service discovery and routing layer. How does this differ from DNS?

I'm not familiar with Consul specifically, but service meshes I am familiar with give you a lot more than just mapping of service names to dynamic IPs. They bundle things like TLS termination, reverse proxying, network policy enforcement, automatic certificate provisioning and key rotation, firewalls potentially aware of anything from layer 3 to layer 7.

Where I think companies are going about this wrong sometimes is thinking having their networks be software-defined and handled by some all-in-one product suite means they no longer need network engineers and can they just rely entirely on application developers who specialize in general programming and fall back to vendor support contracts if anything gets too confusing.

Admittedly, there's some self interest speaking there, because I work as a consulting engineer for one of these vendors where we go way beyond "support" to embed full time in external product teams with a dependency on our products (though we also offer basic support for companies that think they can get away with it).

But to my mind, no software suite can let you get away with not needing any kind of IT ops at all. Smaller companies may assess that the risk is worth it to focus solely on product, but as far as I can tell, Roblox is not a small company (or shouldn't be, given the traffic scale of their platform).

Re: An Update on Our Outage

#147

Post by bsenftner »

ksec wrote: post #138
slg wrote: post #33

My heart goes out to the people who likely had to work crazy hours to fix this, but it really is wild that it was down for so long. What was the last service of this size that was down for 4 days? That is a failure in architecture that goes way beyond whatever the specific cause was here. That post-mortem is going to be a doozy.

I wonder if Engineers from Roblox are now worth a little more just because they have this experience. I also think it is time that people should take a look again at Chaos Engineering [1] from Netflix. It is sometimes ironic that the best technology often comes from companies that aren't a technology company at all. [1] https://principlesofchaos.org

And I wonder if their tech management is worth less for this clusterf** to happen during their oversight?

Re: An Update on Our Outage

#148

Post by 0xbadcafebee »

jpgvm wrote: post #125
sofixa wrote: post #121

Earlier quoted context omitted.

I don't know, anecdotally it seems to me that Consul is more popular than Zookeeper. But yeah, you should go with the one you know best, especially at any sort of scale. Vault is the most popular secrets manager out there, and has a lot of advantages over something like Kubernetes secrets ( which aren't even encrypted). Nomad is a bit obscure but IMHO it seems to be gaining momentum.

ZK is vastly more popular, it's just much less sexy so people don't write about it. Every production Kafka, Spark, Hadoop, Pulsar, etc cluster is using ZK. Just the Kafka and Hadoop clusters alone probably allows it to rival etcd in footprint. Consul on the other hand isn't used widely outside of some startups and a few OSS systems like Grafana Cortex/Loki. When it comes to which one has the most production hours und…

Consul is at least as widely used as ZK now seeing as anyone who wants to run a distributed system on Hashi tools needs to use it. It's much better designed, and easier to operate, than ZK. ZK, Solr, and other apache projects are pretty terrible in comparison.

Re: An Update on Our Outage

#149

Post by bsenftner »

keyle wrote: post #82
slg wrote: post #33

My heart goes out to the people who likely had to work crazy hours to fix this, but it really is wild that it was down for so long. What was the last service of this size that was down for 4 days? That is a failure in architecture that goes way beyond whatever the specific cause was here. That post-mortem is going to be a doozy.

In the game industry, working yourself to death for 4 days, that's called a hot fix ;)

And that is after you've been working 6 days a week for the last 9 months. As an EA spouse survivor - fuck that industry.

Re: An Update on Our Outage

#150

Post by 0xbadcafebee »

strictfp wrote: post #145
jpgvm wrote: post #112

Earlier quoted context omitted.

Generally you need them even when running in a "Cloud" of some description. Namely because they provide distributed locking primitives at a speed that can't be matched by any other mechanism. If you only need service discovery you can probably get away with whatever you platform provides (EC2 API, k8s API, hell DNS works, etc), similarly if you only need slow master election there are alternatives there too (DynamoDB…

Having built many distributed applications I would hesitate to use ZK/consul directly, mostly since your cloud provider is already providing most type of primitives directly or indirectly in the services they provide.

But most people use them because they're required by something else, like Vault, or Solr, or some other tool built to require a particular distributed key value store.
Post reply on HN