Lessons Learned from Twenty Years of Site Reliability Engineering
1–10 of 128 posts
Re: Lessons Learned from Twenty Years of Site Reliability Engineering
#2[deleted]
Re: Lessons Learned from Twenty Years of Site Reliability Engineering
#3"We once narrowly missed a major outage because the engineer who submitted the would-be-triggering change unplugged their desktop computer before the change could propagate". Sorry, what?
Re: Lessons Learned from Twenty Years of Site Reliability Engineering
#4"We once narrowly missed a major outage because the engineer who submitted the would-be-triggering change unplugged their desktop computer before the change could propagate". Sorry, what?
I thought the same.
Re: Lessons Learned from Twenty Years of Site Reliability Engineering
#5"We once narrowly missed a major outage because the engineer who submitted the would-be-triggering change unplugged their desktop computer before the change could propagate". Sorry, what?
The change was being orchestrated from their desktop, and they noticed thing were going sideways, so they unplugged their desktop to stop the deployment. Aka "pressed the big red button".
Re: Lessons Learned from Twenty Years of Site Reliability Engineering
#6"We once narrowly missed a major outage because the engineer who submitted the would-be-triggering change unplugged their desktop computer before the change could propagate". Sorry, what?
Yeah, interesting tidbit. It might sound insane today that one engineer's desktop computer could cause such an outage. But that was probably more commonplace 20 years ago and even today in smaller orgs.
Re: Lessons Learned from Twenty Years of Site Reliability Engineering
#7This is a great writeup, and very broadly applicable. I don't see any "this would only apply at Google" in here.
> COMMUNICATION CHANNELS! AND BACKUP CHANNELS!! AND BACKUPS FOR THOSE BACKUP CHANNELS!!!
Yes! At Netflix, when we picked vendors for systems that we used during an outage, we always had to make sure they were not on AWS. At reddit we had a server in the office with a backup IRC server in case the main one we used was unavailable.
And I think Google has a backup IRC server on AWS, but that might just apocryphal.
Always good to make sure you have a backup side channel that has as little to do with your infrastructure as possible.
Re: Lessons Learned from Twenty Years of Site Reliability Engineering
#8"We once narrowly missed a major outage because the engineer who submitted the would-be-triggering change unplugged their desktop computer before the change could propagate". Sorry, what?
At one point I had to run a script on a substantial portion of their server fleet (like hundreds of thousands machines) and I remember I ran it with a pssh-style utility from desktop (was 10y ago so dunno if they still use this). It was surprisingly quick to do it this way. Could’ve been something like that
Re: Lessons Learned from Twenty Years of Site Reliability Engineering
#9"We once narrowly missed a major outage because the engineer who submitted the would-be-triggering change unplugged their desktop computer before the change could propagate". Sorry, what?
It always cracks me up how Google is simultaneously the most web-based company in the world, but their internal political landscape was (Infra, Search, Ads) > everything else. This leads to infra swe writing stupid CLIs all day, rather than having any literal buttons. Things were changing a lot by the time I left though.
I do think Google should be more open about their internal outages. This one in particular was very famous internally.
Re: Lessons Learned from Twenty Years of Site Reliability Engineering
#10This is a great writeup, and very broadly applicable. I don't see any "this would only apply at Google" in here. > COMMUNICATION CHANNELS! AND BACKUP CHANNELS!! AND BACKUPS FOR THOSE BACKUP CHANNELS!!! Yes! At Netflix, when we picked vendors for systems that we used during an outage, we always had to make sure they were not on AWS. At reddit we had a server in the office with a backup IRC server in case the main one…
Afair Google just ran irc on their corp network which was completely separate from prod so I wouldn’t be surprised if it was in a small server room in the office somewhere.
> very broadly applicable. I don't see any "this would only apply at Google" in here.
One thing I haven’t even heard of anyone else doing was production panic rooms - a secure room with backup vpn to prod