Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

An update on Sunday’s service disruption

cloud.google.com

Post reply on HN
Print view View original

51–60 of 220 posts

Re: An update on Sunday’s service disruption

#51

Post by singron »

jameslk wrote: post #5

> In essence, the root cause of Sunday’s disruption was a configuration change I feel like I hear about config changes breaking these cloud hosts so often it might as well be a meme. Is there a reason why it's usually configurations to blame vs code, hardware, etc?

It's common practice in many organizations to effectively deploy new code with configuration changes. E.g. you write new code that is initially disabled then enable it with a configuration change. Since the code deploy didn't fail, you get a false confidence in the code and the configuration change seems like a safe "flag flip".

Re: An update on Sunday’s service disruption

#52

Post by richardw »

In these outages there's so often someone dicking with the system. Config change, upgrade, etc. I once asked for a "stable" version of app engine that they largely left alone. Not sure if that's possible or would be better - it's likely that the vast majority of upgrades are bulletproof. But still...there's danger in fiddling.

Re: An update on Sunday’s service disruption

#54

Post by keypusher »

flukus wrote: post #27
jameslk wrote: post #5

> In essence, the root cause of Sunday’s disruption was a configuration change I feel like I hear about config changes breaking these cloud hosts so often it might as well be a meme. Is there a reason why it's usually configurations to blame vs code, hardware, etc?

Well you can't just change code, it has to get reviewed, it has to go to QA, it has to go to UAT, it has to get signed off in triplicate by all the major stake holders. Configuration changes are easy though, they don't have to go through all these error prevention steps, we can just have our less technical support staff make configuration changes live in production. In fact we'll build them a DSL so we never have to…

A modern ops team should be sending configuration changes through the same code review process application code goes through. Infrastructure changes in something like Terraform, configuration changes in Puppet/Chef/etc, these are deployed and tested in dev, staging, prod.

Re: An update on Sunday’s service disruption

#55

Post by IntricateJake »

nodesocket wrote: post #11

It feels a little strange and calculated since the outage impacted mostly US based regions, that Google released this update late at night (relative to Central and Eastern time). Wouldn't it make more sense to release it tomorrow, Tuesday at like 11am Eastern (8am Pacific) for full transparency for the affected companies?

Google was clearing their data collection servers due to an NSA audit

Re: An update on Sunday’s service disruption

#56

Post by [deleted] »

snewman wrote: post #29

This is a surprisingly vague postmortem. No timeline, no specific identification of affected regions. And no explanation of why a configuration change that was (apparently) made with a single command required so much effort to undo, or why repair efforts were hampered when (again, apparently) the network was successfully prioritizing high-priority traffic. Even for a public postmortem, this seems pretty weak.

[deleted]

Re: An update on Sunday’s service disruption

#57

Post by mkl »

murat124 wrote: post #43

This seems a quick write up from a manager to the managers that simply says how big they are and that they are sorry. I doubt the public will ever see a technical postmortem. Still there are great lessons in this incident for them as much as for all SREs around the world who struggled during the incident. I for one wouldn't want to rely on a global load balancer which I know now that can not survive a regional outage…

> I doubt the public will ever see a technical postmortem.

Why not? We usually do, e.g. https://news.ycombinator.com/item?id=17569069 from 10 months ago.

Re: An update on Sunday’s service disruption

#58

Post by theevilsharpie »

JMTQp8lwXL wrote: post #38
snewman wrote: post #29

This is a surprisingly vague postmortem. No timeline, no specific identification of affected regions. And no explanation of why a configuration change that was (apparently) made with a single command required so much effort to undo, or why repair efforts were hampered when (again, apparently) the network was successfully prioritizing high-priority traffic. Even for a public postmortem, this seems pretty weak.

The lack of transparency makes me want to consider other cloud providers. All providers will have outages -- that's a reality I can live with -- but I will prioritize the ones who are the most forthcoming in their statuses and explanations into failures.

Google kept their status page up to date as the outage was progressing, and now (the day after the outage), they've provided an apology and a preliminary explanation of what happened.

If that's not sufficient, what more are you looking for, and what other large cloud providers consistently meet that standard?

Re: An update on Sunday’s service disruption

#59

Post by summerlight »

jameslk wrote: post #5

> In essence, the root cause of Sunday’s disruption was a configuration change I feel like I hear about config changes breaking these cloud hosts so often it might as well be a meme. Is there a reason why it's usually configurations to blame vs code, hardware, etc?

Typically configuration changes tend to have much more impacts compared to implementation codes at the same line counts hence its consequences are less predictable. Also, it usually coordinates multiple systems at once, so it is much harder to test especially when it comes to a capacity problem caused by complex interactions among multiple systems. Canary or gradual roll-out cannot reliably catch this class of problems, since it's not really visible in a small scale.

Re: An update on Sunday’s service disruption

#60

Post by photonios »

nodesocket wrote: post #40
ehsankia wrote: post #39

Earlier quoted context omitted.

It didn't say it was a small impact, but that it impacted a small number of users. If you were one of those users, it was high impact for you, but the number of impacted users was small.

Fair, but I have friends on the west coast who were similarly impacted, while I am in the South. So it seems inaccurate to say it impacted a small number of users. Maybe compared to world wide, but I would imagine the US is the biggest market.

> However, for users who rely on services homed in the affected regions, the impact was substantial, particularly for services like YouTube or Google Cloud Storage which use large amounts of network bandwidth to operate.

The post admits that. It clearly says that the impact on users in affected regions was significant but that some regions were barely affected. It would've been nice if they mentioned what regions. But beside that, what's the problem?

Post reply on HN