Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

An update on Sunday’s service disruption

cloud.google.com

Post reply on HN
Print view View original

161–170 of 220 posts

Re: An update on Sunday’s service disruption

#161

Post by vikramkr »

JMTQp8lwXL wrote: post #38
snewman wrote: post #29

This is a surprisingly vague postmortem. No timeline, no specific identification of affected regions. And no explanation of why a configuration change that was (apparently) made with a single command required so much effort to undo, or why repair efforts were hampered when (again, apparently) the network was successfully prioritizing high-priority traffic. Even for a public postmortem, this seems pretty weak.

The lack of transparency makes me want to consider other cloud providers. All providers will have outages -- that's a reality I can live with -- but I will prioritize the ones who are the most forthcoming in their statuses and explanations into failures.

This isnt the postmortem though. It says they're still working on that.

Re: An update on Sunday’s service disruption

#162

Post by sokoloff »

JMTQp8lwXL wrote: post #38
snewman wrote: post #29

This is a surprisingly vague postmortem. No timeline, no specific identification of affected regions. And no explanation of why a configuration change that was (apparently) made with a single command required so much effort to undo, or why repair efforts were hampered when (again, apparently) the network was successfully prioritizing high-priority traffic. Even for a public postmortem, this seems pretty weak.

The lack of transparency makes me want to consider other cloud providers. All providers will have outages -- that's a reality I can live with -- but I will prioritize the ones who are the most forthcoming in their statuses and explanations into failures.

I don't experience this as anything like a lack of transparency.

The incident was less than 2 days ago, is resolved, and we have a preliminary report from the "VP, 24x7", which is easily digestible by the average GCP customer with more details undoubtedly to come.

Re: An update on Sunday’s service disruption

#163

Post by gregdoesit »

milofeynman wrote: post #103
gregdoesit wrote: post #18

I wish the Google team shared or could more details on the incident: like the timeline, how long the total outage took and what preventative actions they are taking to fix a similar issue from happening at a systemic level, or mitigation to be substantially faster next time. This update feels like it just shares the root cause at a high level (configuration change) and norms much else.

I found this write up with some metrics/ timeline on Twitter https://lightstep.com/blog/googles-june-2nd-outage-their-sta... I still don't think it's the full picture. But better than nothing

Thank you!

Re: An update on Sunday’s service disruption

#164

Post by the-rc »

jrockway wrote: post #143

This answered all the questions I had. I was really racking my brain on what one system at Google could go down to cause this much damage, but it makes perfect sense that bandwidth becoming unavailable and everything in the "default" or "bulk" traffic class being dropped would do it. The real question is whether the fix will be to not reduce bandwidth accidentally, or to upgrade customer traffic to a higher QoS class…

I speculated it was something to do with the SDN, among others:

https://news.ycombinator.com/item?id=20078433

It was unlikely to be fiber or a router failing, because there's enough redundancy at all sorts of levels (usually N+2 or better). Unless, that is, some nation state had been cutting multiple fibers at once.

This had the hallmark of some system blowing up, as you said. When it comes to QoS, it gets tricky. Gmail's frontend traffic should be at the highest priority, of course. But what about the replication traffic between your mailbox homes? What if a top level layer stalls or chokes when replication lags too much behind?

It's easier for stateless or less stateful systems like web search.

Re: An update on Sunday’s service disruption

#165

Post by OJFord »

killjoywashere wrote: post #137
awalton wrote: post #136

Earlier quoted context omitted.

Perimortem?

In pathology we call issue a "preliminary autopsy report" within the first 48 hours and a "final autopsy report" within 30-60 days. A final report may be followed by additional addenda (new information, e.g. toxicology reports) and amendments (significant changes to the original report, e.g. "I was wrong")

Yeah, the issue really is that post-mortem's been adopted as a noun.

This is a de facto post-mortem (allowing the adoption of 'mortem' to mean problem, but not the phrase as noun) just perhaps not the full analysis it's been taken to imply.

Re: An update on Sunday’s service disruption

#166

Post by dsfyu404ed »

jrockway wrote: post #143

This answered all the questions I had. I was really racking my brain on what one system at Google could go down to cause this much damage, but it makes perfect sense that bandwidth becoming unavailable and everything in the "default" or "bulk" traffic class being dropped would do it. The real question is whether the fix will be to not reduce bandwidth accidentally, or to upgrade customer traffic to a higher QoS class…

>Engineers building apps that depend on that know the limitations.

As someone who works at a slightly smaller tech company with of similar age with similar infrastructure I assure you this is not the case. Engineers are building things that rely on other things that rely on other things. There's a point where people don't know what their dependencies are.

I wouldn't be surprised if nobody actually knew there was customer traffic in this class until this happened.

Re: An update on Sunday’s service disruption

#167

Post by gamegod »

markphip wrote: post #149

Is it just me or is this lacking any acknowledgment of the impact it had on GCE and all of the third parties that were impacted by this. They make it sound like a few people could not watch YouTube videos and even fewer people had some email disruption but this outage had a lot more impact than that. As just one example, a huge number of Shopify sites were impacted by this as were I am sure a number of other SaaS bus…

The network became congested, and our networking systems correctly triaged the traffic overload and dropped larger, less latency-sensitive traffic in order to preserve smaller latency-sensitive traffic flows...

Overall, YouTube measured a 10% drop in global views during the incident...

So what I'm hearing is that while Google Cloud Pub/Sub was down for hours, crippling my SaaS business, Google was prioritizing traffic to cat videos.

It's good to know Google considers GCP traffic neither important, nor urgent.

Re: An update on Sunday’s service disruption

#168

Post by grogers »

One of my favorite patterns for updating configuration in-band I learned from Juniper routers. When you enact a new configuration you can have it automatically rollack after some period of time unless you confirm the configuration. Often the pattern is to intentionally have it roll back after a short period (e.g. one minute), then again after a longer period, (e.g. 10 minutes) and the on the last time you make it permanent. I feel like all configuration systems should allow for a similar mechansim

Re: An update on Sunday’s service disruption

#169

Post by ineedasername »

dsfyu404ed wrote: post #166
jrockway wrote: post #143

This answered all the questions I had. I was really racking my brain on what one system at Google could go down to cause this much damage, but it makes perfect sense that bandwidth becoming unavailable and everything in the "default" or "bulk" traffic class being dropped would do it. The real question is whether the fix will be to not reduce bandwidth accidentally, or to upgrade customer traffic to a higher QoS class…

>Engineers building apps that depend on that know the limitations. As someone who works at a slightly smaller tech company with of similar age with similar infrastructure I assure you this is not the case. Engineers are building things that rely on other things that rely on other things. There's a point where people don't know what their dependencies are. I wouldn't be surprised if nobody actually knew there was cust…

I've never worked in this type of operation, can you shed some light? I would have thought there'd be some type of documentation of the dependency hierarchy for change request checklists. Or are such things not always quite as comprehensive ( or not possible to have such complex interdependencies be comprehensively documented) ?
Post reply on HN