Putting aside the content of the page, can we talk about the design of the page? It has this HUGE hero image that pushes the content down below the fold on my 4k monitor.
An Update on Our Outage
51–60 of 235 posts
Re: An Update on Our Outage
#52Earlier quoted context omitted.
What I love most then is that its the largest and it doesn't matter! In this market they will become an even larger company because competency has no correlation to how much money subscribers/advertisers will pay, and none of that has any correlation to what investors will pay! Anybody optimizing for competence is exchanging time for food and shelter and just gets to be reminded of the much harder game they are playi…
Maybe your definition of competence is wrong? A lot of people can build systems with some number of 9s of availability, but how many people can build a 50B company?
One is luck and the other is engineering. From the looks of it, this is a high luck, low competency endeavor.
Re: An Update on Our Outage
#53Someone on here posted the other day that someone they knew at Roblox said it was their secret store that became overloaded. Presumably a Hashicorp Vault type service (or something similar.) This update appears to support that claim. That's definitely a problem you only get at scale.
Not far off. Having chatted with friends from the company, consul failed due to a issue with streaming which was introduced in 1.9 ( https://www.hashicorp.com/blog/announcing-hashicorp-consul-1... ) which caused it to have massively reduced tps in certain circumstances. No functional consul meant no vault, no consul/vault meant no nomad, no nomad meant no application servers. Furthermore, the services were down for l…
> no consul/vault meant no nomad, no nomad meant no application servers.
Not super familiar with nomad but it seems like that would not necessarily follow. For example if etcd goes fully dark in kubernetes cluster things will mostly continue running for a while unless something also crashes the servers
Re: An Update on Our Outage
#54Earlier quoted context omitted.
As someone not familiar with HashiCorp's products, could you give a super high level overview of what consul, vault and nomad are?
Consul is a service mesh, vault is secret storage and nomad is for workload orchestration.
So relating to things I do know about, consul is like cloud infra, vault is a service for secret storage and nomad is like Kubernetes?
Re: An Update on Our Outage
#55Earlier quoted context omitted.
Not far off. Having chatted with friends from the company, consul failed due to a issue with streaming which was introduced in 1.9 ( https://www.hashicorp.com/blog/announcing-hashicorp-consul-1... ) which caused it to have massively reduced tps in certain circumstances. No functional consul meant no vault, no consul/vault meant no nomad, no nomad meant no application servers. Furthermore, the services were down for l…
As someone not familiar with HashiCorp's products, could you give a super high level overview of what consul, vault and nomad are?
Consul is a service mesh. It’s your dynamic service discovery and routing layer. You have systems dynamically allocated in a cluster, they need to reach other systems, you ask consul where they are.
Vault is secrets and management. Put secret strings in, get secret strings out (if permitted). Most apps need secrets of some type, and vault is normally discovered via consul.
Nomad is an application/workload scheduler. You tell it what you need to run and what their memory/cpu requirements are and it finds a space in your physical infrastructure to run that. The apps it runs normally needs secrets from vault and communicate with services discovered via consul.
They all are well integrated and build off of each other, kind of like layers an onion where your app services are the outer layer of the onion. Consul failing this badly is like the core of the onion going rotten. There’s not much saving it at that point, you need to grow a new onion from the inside out, but that takes time.
Re: An Update on Our Outage
#56Earlier quoted context omitted.
Not far off. Having chatted with friends from the company, consul failed due to a issue with streaming which was introduced in 1.9 ( https://www.hashicorp.com/blog/announcing-hashicorp-consul-1... ) which caused it to have massively reduced tps in certain circumstances. No functional consul meant no vault, no consul/vault meant no nomad, no nomad meant no application servers. Furthermore, the services were down for l…
> no consul/vault meant no nomad, no nomad meant no application servers. Not super familiar with nomad but it seems like that would not necessarily follow. For example if etcd goes fully dark in kubernetes cluster things will mostly continue running for a while unless something also crashes the servers
True for long running services, but not necessarily ephemeral workloads. Those need a system to schedule them or else once they complete they just stop running.
Also, as long running services crash or reboot naturally they need to be rescheduled or else your cluster slowly dies, and as your cluster size increases the mtbf decreases and the need to reschedule workloads continually increases.
Re: An Update on Our Outage
#57My heart goes out to the people who likely had to work crazy hours to fix this, but it really is wild that it was down for so long. What was the last service of this size that was down for 4 days? That is a failure in architecture that goes way beyond whatever the specific cause was here. That post-mortem is going to be a doozy.
Not necessarily directed at roblox but honestly I’m surprised this doesn’t happen more often given how many teams I see run software they don't understand or don’t even have access to its source code.
Edit: but yeah must’ve been tough 72+hrs i hope their version of reliability team can use this to bash the support they need out of the management and not get scapegoated instead
Re: An Update on Our Outage
#58Earlier quoted context omitted.
Not far off. Having chatted with friends from the company, consul failed due to a issue with streaming which was introduced in 1.9 ( https://www.hashicorp.com/blog/announcing-hashicorp-consul-1... ) which caused it to have massively reduced tps in certain circumstances. No functional consul meant no vault, no consul/vault meant no nomad, no nomad meant no application servers. Furthermore, the services were down for l…
As someone not familiar with HashiCorp's products, could you give a super high level overview of what consul, vault and nomad are?
I'd recommend reading the official Hashicorp website for that, but the gist is that nomad is another way of deploying apps in containers (similar to kubernetes), consul is how apps find other apps to talk to (kinda like DNS but with health checking built-in so they don't get stale info), and vault is how apps retrieve the credentials they need to connect to other services.
Re: An Update on Our Outage
#59Earlier quoted context omitted.
Consul is a service mesh, vault is secret storage and nomad is for workload orchestration.
So relating to things I do know about, consul is like cloud infra, vault is a service for secret storage and nomad is like Kubernetes?
Consul is like etcd but has some extra features built in like service discovery and l7 proxy so they market it as full blown service mesh a la istio, the other two are spot on
Re: An Update on Our Outage
#60Earlier quoted context omitted.
Maybe your definition of competence is wrong? A lot of people can build systems with some number of 9s of availability, but how many people can build a 50B company?
One is luck and the other is engineering. From the looks of it, this is a high luck, low competency endeavor.
And Roblox is already well beyond the luck threshold