61–70 of 235 posts
My 3 youngest kids, aged 9, 9, and 8 told me Roblix were having an extended outage, world wide. Good job to the team getting everything back up. Been there, done that. #hugops
That must have been quite a year in the White household!
Earlier quoted context omitted.
As someone not familiar with HashiCorp's products, could you give a super high level overview of what consul, vault and nomad are?
Consul is a service mesh. It’s your dynamic service discovery and routing layer. You have systems dynamically allocated in a cluster, they need to reach other systems, you ask consul where they are. Vault is secrets and management. Put secret strings in, get secret strings out (if permitted). Most apps need secrets of some type, and vault is normally discovered via consul. Nomad is an application/workload scheduler.…
Earlier quoted context omitted.
Not far off. Having chatted with friends from the company, consul failed due to a issue with streaming which was introduced in 1.9 ( https://www.hashicorp.com/blog/announcing-hashicorp-consul-1... ) which caused it to have massively reduced tps in certain circumstances. No functional consul meant no vault, no consul/vault meant no nomad, no nomad meant no application servers. Furthermore, the services were down for l…
As someone not familiar with HashiCorp's products, could you give a super high level overview of what consul, vault and nomad are?
Consul: It's used to help manage cloud networking so that your application doesn't have to worry about IP addresses, datacenter locations, punching through firewalls, etc. Think of the situation where you have a zillion microservices talking to each other on different machines- it makes it easier for them to find each other. It also includes a distributed key-value store
Vault: If you've used a password manager in your browser, imagine that but distributed and on steroids. You can use it to share credentials with groups. It also has some APIs to help with encryption/decryption and includes a key-value store
Nomad: Roughly similar to Kubernetes (although with good support for non-containerized software). It's used to orchestrate software. For example when you have a bunch of programs you want to run 24/7 , you can specify what machines they should run on (for example at the datacenter/region level), what resources to give the programs, how to handle hardware failures, etc.
Earlier quoted context omitted.
What I love most then is that its the largest and it doesn't matter! In this market they will become an even larger company because competency has no correlation to how much money subscribers/advertisers will pay, and none of that has any correlation to what investors will pay! Anybody optimizing for competence is exchanging time for food and shelter and just gets to be reminded of the much harder game they are playi…
Maybe your definition of competence is wrong? A lot of people can build systems with some number of 9s of availability, but how many people can build a 50B company?
Didn't realize I struck a cord here
I’m literally making fun of the people that would try to draw this distinction and … you showed up?
Huh
Sometimes I forget that the market reality changes faster than cultural conditioning
Change faster
Yet they still take a 30% cut on all transactions that go to content creators / developers
Earlier quoted context omitted.
That or DNS. ;-)
Or maybe BGP >:)
Earlier quoted context omitted.
Thank you for posting the text of the blog post. However, I don't understand your TLDR. They clearly were able to identify the root cause, and that is what allowed them to restore service.
I mean this is honestly a PR/lawyer blog post. > A core system in our infrastructure became overwhelmed, prompted by a subtle bug in our backend service communications while under heavy load. Translation Something important went down, we couldn't trace the bug. >This was not due to any peak in external traffic or any particular experience. Translation We're not blaming Chipotle for this > most services at Roblox were…
> Rather the failure was caused by the growth in the number of servers in our datacenters.
Their config server ran out of sockets and screwed up everything. Or something like that. Too many servers = nothing works has many ways to unfold and isn't obvious the first time.
Earlier quoted context omitted.
What does that tend to look like? Do you have any examples?
In my experience, it tends to be that you find out one of your components has some tipping point as you scale out horizontally where everything goes to hell. Random example: you scale a service horizontally and suddenly postgres disk space usage goes from "this line looks horizontal" to "we just used 2 years' worth of disk use growth in 40 seconds". And the postmortem is basically like "yep can't really blame anyone…
Because the business had been around for a while, there was two separate ways to deploy and manage apps: old and new.
We reconfigured how code was deployed within and between racks for increase resistance to racks or even whole colos dying. This change was made in the new deployment system. And all was well for most of a year. Software and services gradually migrated from old to new as people had the time and inclination.
The new deployment also packed services more tightly, so most computers kept copies of most binaries on them.
Then we crossed one of those tipping points. On a code update, several thousand servers across the world demanded a small set of binaries from a distributed data node via the newer deployment method. Which would be fine, except we had degraded service from a couple TOR routers due to a separate bug. And so the distributed data nodes hosting this particular set of binaries were less "distributed" and more "just one poor about to be overloaded machine and network."
So a few thousand machines demanded a couple hundred megs nearly simultaneously. And when they didn't get it in a timely fashion, all triggered a rollback. Simultaneously. Which demanded another couple hundred mb of code. Simultaneously. And then the freakouts started.
And because other deploys were in flight, there was more than one service deploy ongoing. And they all started fighting each other for bandwidth.
> A key value at Roblox is “Respect the Community,” Yet they still take a 30% cut on all transactions that go to content creators / developers
Saying this as someone who doesn’t know much about Roblox but does give 15-30% to Apple and Google. Google definitely doesn’t respect their developers. Apple kind of does.
> A key value at Roblox is “Respect the Community,” Yet they still take a 30% cut on all transactions that go to content creators / developers