13:20 PST = 16:20 EST
DNS Outage Post Mortem
11–16 of 16 posts
Re: DNS Outage Post Mortem
#12Who in the right mind would schedule a critical infrastructure upgrade during the day?
Firstly, deploying during the day is good because if something goes wrong, the entire team is physically present to deal with it. Second, Github is used worldwide, which means 24 hours a day, so there isn't a "night" to deploy during.
Re: DNS Outage Post Mortem
#13These are some of the corner cases that are put on the back burner en route to delivering an MVP. Just like you don't do early optimization of an infrastructure, you almost never enumerate all possible issues that can crop up under a less than ideal situation. This isn't a GitHub only issue but rather one that would affect all quick-to-launch startups (most). What I'm learning from this is that one needs to regularly…
+1. This is invaluable. This falls in line within a larger frame of thinking -- "immutable infrastructure." Schedule time to regularly provision your entire stack from ground up without any of the caching optimizations and have it run in production.
With tools like Chef & AWS CloudFormation, there shouldn't be an excuse.
Re: DNS Outage Post Mortem
#14This is why i like Chef...i feel there are tools out there to test your code better....FoodCritic...ChefSpec...Test Kitchen...before rolling to production and having to validate machines in production....ouch
Re: DNS Outage Post Mortem
#15What a seriously dumb outage to have. I'm still confused about it after reading the the RFO.
Re: DNS Outage Post Mortem
#16"an initial verification led us to believe the changes had been rolled out successfully" I would love more detail in the type II error in this validation step, and is worth exploring deeper. What was the verification step? Why did it not detect the issue? What review process was used for the verification step? While the failed verification step is not the root cause, having good safety checks are the most important p…
I was surprised that the total amount of time between rolling out to the first servers, waiting, verifying, and then rolling out to the second set of servers was a whopping nine minutes.
Maybe I'm just too careful (perhaps because I've seen it happen before) but I prefer to wait a helluva lot longer than that for verification.
And perhaps it's because I dealt with Microsoft Active Directory so much in the past but I am extremely careful when it comes to DNS. If there's one thing that'll screw up your entire environment (especially in an AD-based network), it's broken DNS.