The amazing thing is how many times they've had this exact outage or a close relative of it and Ben Treynor still gets to keep his job.
Come on you guys. Plesae don't cross into personal attack.
An update on Sunday’s service disruption
31–40 of 220 posts
Re: An update on Sunday’s service disruption
#32> In essence, the root cause of Sunday’s disruption was a configuration change I feel like I hear about config changes breaking these cloud hosts so often it might as well be a meme. Is there a reason why it's usually configurations to blame vs code, hardware, etc?
Almost all outages are due to changes. Bugs in code usually surface pretty quickly, and hardware issues don't pose a problem in such large-scale infrastructure. Changes causing outages is par for the course. That's why the industry standard is now Infrastructure-as-Code and Immutable Infrastructure: you try to prevent anything from changing in a way that will later break, and manage it in ways that will be easier to fix when it does break.
Tools like Terraform are popular today because they allow the planning and staging of changes across complex services. They're still pretty limited, but mapping out dependencies and simulating changes can surface errors before you run into them, thus making it less necessary to perform a rollback. But unexpected problems still happen, which is why you need to test your rollbacks, and intentionally stress random parts of your system to discover unknown bottlenecks. Part of the purpose for stress testing is to have a realistic idea of what kind of capacity you will really have under different conditions. But it's also nearly impossible to accurately stress test production systems without consequences.
There are ways for Google to look for change issues, and they probably have lots of safeguards in place, but we don't know what they actually do to test changes. Some of their postmortems have pointed at a lack of stringent change control procedures. Hopefully they will practice what they preach (open/blameless postmortems) and share more details soon.
Re: An update on Sunday’s service disruption
#33I wish the Google team shared or could more details on the incident: like the timeline, how long the total outage took and what preventative actions they are taking to fix a similar issue from happening at a systemic level, or mitigation to be substantially faster next time. This update feels like it just shares the root cause at a high level (configuration change) and norms much else.
> Google Cloud Storage measured a 30% reduction in traffic With things like these the monetary value is so huge their legal team will never allow them to give details. More detail, more chance of lawsuits
I've seen much more detail in other postmortems from major cloud providers. (AWS and Azure definitely, and I think Google as well.)
Re: An update on Sunday’s service disruption
#34> Overall, YouTube measured a 10% drop in global views during the incident, while Google Cloud Storage measured a 30% reduction in traffic. Approximately 1% of active Gmail users had problems with their account; while that is a small fraction of users
G suite failed to sync e-mail. My Nest app was completely down via iPhone. Google Home when asked for the weather in Nashville responded with "I can't help with that...", and a GCE MySQL instance in us-west2 (Los Angeles) was down for 3 hours for me. Not a small impacting incident.
Re: An update on Sunday’s service disruption
#35This is a surprisingly vague postmortem. No timeline, no specific identification of affected regions. And no explanation of why a configuration change that was (apparently) made with a single command required so much effort to undo, or why repair efforts were hampered when (again, apparently) the network was successfully prioritizing high-priority traffic. Even for a public postmortem, this seems pretty weak.
> With all services restored to normal operation, Google’s engineering teams are now conducting a thorough post-mortem ...
This is not the post-mortem, that is still to come.
Re: An update on Sunday’s service disruption
#36I wish the Google team shared or could more details on the incident: like the timeline, how long the total outage took and what preventative actions they are taking to fix a similar issue from happening at a systemic level, or mitigation to be substantially faster next time. This update feels like it just shares the root cause at a high level (configuration change) and norms much else.
> With all services restored to normal operation, Google’s engineering teams are now conducting a thorough post-mortem...
Still to come.
Re: An update on Sunday’s service disruption
#37This is a surprisingly vague postmortem. No timeline, no specific identification of affected regions. And no explanation of why a configuration change that was (apparently) made with a single command required so much effort to undo, or why repair efforts were hampered when (again, apparently) the network was successfully prioritizing high-priority traffic. Even for a public postmortem, this seems pretty weak.
> With all services restored to normal operation, Google’s engineering teams are now conducting a thorough post-mortem ... This is not the post-mortem, that is still to come.
The postmortem is currently only visible to Googlers.
Re: An update on Sunday’s service disruption
#38This is a surprisingly vague postmortem. No timeline, no specific identification of affected regions. And no explanation of why a configuration change that was (apparently) made with a single command required so much effort to undo, or why repair efforts were hampered when (again, apparently) the network was successfully prioritizing high-priority traffic. Even for a public postmortem, this seems pretty weak.
The lack of transparency makes me want to consider other cloud providers. All providers will have outages -- that's a reality I can live with -- but I will prioritize the ones who are the most forthcoming in their statuses and explanations into failures.
Re: An update on Sunday’s service disruption
#39> Overall, YouTube measured a 10% drop in global views during the incident, while Google Cloud Storage measured a 30% reduction in traffic. Approximately 1% of active Gmail users had problems with their account; while that is a small fraction of users G suite failed to sync e-mail. My Nest app was completely down via iPhone. Google Home when asked for the weather in Nashville responded with "I can't help with that...…
It didn't say it was a small impact, but that it impacted a small number of users. If you were one of those users, it was high impact for you, but the number of impacted users was small.
Re: An update on Sunday’s service disruption
#40> Overall, YouTube measured a 10% drop in global views during the incident, while Google Cloud Storage measured a 30% reduction in traffic. Approximately 1% of active Gmail users had problems with their account; while that is a small fraction of users G suite failed to sync e-mail. My Nest app was completely down via iPhone. Google Home when asked for the weather in Nashville responded with "I can't help with that...…
It didn't say it was a small impact, but that it impacted a small number of users. If you were one of those users, it was high impact for you, but the number of impacted users was small.
Fair, but I have friends on the west coast who were similarly impacted, while I am in the South. So it seems inaccurate to say it impacted a small number of users. Maybe compared to world wide, but I would imagine the US is the biggest market.