11–20 of 128 posts
E.g. do you use db-based "feature flags"? What do you do then if the DB itself is overloaded, or the API through which you access the DB? Or do you use static startup flags (e.g. env variables)? How do you ensure these get rolled out quickly enough? Something else entirely?
I'm curious how people approach big red buttons and intentional graceful degradation in practice, and especially how to ensure that these work when the system is experiencing problems. E.g. do you use db-based "feature flags"? What do you do then if the DB itself is overloaded, or the API through which you access the DB? Or do you use static startup flags (e.g. env variables)? How do you ensure these get rolled out q…
... But once your firm starts making guarantees like "five nines uptime," there will be some complexity necessary to devise a system that can continue to be developed and improved while maintaining those guarantees.
"We once narrowly missed a major outage because the engineer who submitted the would-be-triggering change unplugged their desktop computer before the change could propagate". Sorry, what?
Yeah, interesting tidbit. It might sound insane today that one engineer's desktop computer could cause such an outage. But that was probably more commonplace 20 years ago and even today in smaller orgs.
But my favorite incident will forever be the time they had to drill out a safe because they were disaster-testing the password vault system and discovered that the key needed to restore the password vault system was stored in s aafe, the combination for which had been moved into the password vault system. Only with advanced, modern technology can you lock the keys to the safe in the safe itself with so many steps!
I'm curious how people approach big red buttons and intentional graceful degradation in practice, and especially how to ensure that these work when the system is experiencing problems. E.g. do you use db-based "feature flags"? What do you do then if the DB itself is overloaded, or the API through which you access the DB? Or do you use static startup flags (e.g. env variables)? How do you ensure these get rolled out q…
At google we also had to routinely do “backend drains” of particular clusters when we deemed them unhealthy and they had a system to do that quickly at the api/lb layer. At other places I’ve also seen that done with application level flags so you’d do kubectl edit which is obviously less than ideal but worked
"We once narrowly missed a major outage because the engineer who submitted the would-be-triggering change unplugged their desktop computer before the change could propagate". Sorry, what?
It always cracks me up how Google is simultaneously the most web-based company in the world, but their internal political landscape was (Infra, Search, Ads) > everything else. This leads to infra swe writing stupid CLIs all day, rather than having any literal buttons. Things were changing a lot by the time I left though. I do think Google should be more open about their internal outages. This one in particular was ve…
This is a great writeup, and very broadly applicable. I don't see any "this would only apply at Google" in here. > COMMUNICATION CHANNELS! AND BACKUP CHANNELS!! AND BACKUPS FOR THOSE BACKUP CHANNELS!!! Yes! At Netflix, when we picked vendors for systems that we used during an outage, we always had to make sure they were not on AWS. At reddit we had a server in the office with a backup IRC server in case the main one…
This is a great writeup, and very broadly applicable. I don't see any "this would only apply at Google" in here. > COMMUNICATION CHANNELS! AND BACKUP CHANNELS!! AND BACKUPS FOR THOSE BACKUP CHANNELS!!! Yes! At Netflix, when we picked vendors for systems that we used during an outage, we always had to make sure they were not on AWS. At reddit we had a server in the office with a backup IRC server in case the main one…
Also make sure your backup channel can scale. Getting a flood of 10,000+ folks over to a dinky IRC server can knock it over easy. Throttling new joinees isn't a panacea either, since there might be someone critical to get on a channel which throttling can complicate.
Surely 10,000 Users each on average receiving perhaps five 50 byte messages per second (that's a very busy chat room!) is a total bandwidth of 2.5 megabytes per second.
And the CPU time to shuffle messages from one TCP connection to another, or encrypt/decrypt 2.5 megabytes per second should be small. There is no complex parsing involved - it is literally just "forward this 50 byte message into this list of TCP connections".
If they're all internal/authed users, you can hopefully assume nobody is deliberately DoSing the server with millions of super long messages into busy channels too.
This made me think: is SRE a byproduct of a bubble economy of easy money? Why not operate without the significant added expense of SRE teams?
I wonder if SRE will be around 10 years from now, much like Sys Admins and QA testers have mostly disappeared. Instead, many of those functions are performed by software development teams.
If you're using AWS resources, give LocalStack a try for integration testing