An update on Sunday’s service disruption • Hacker News Board

Re: An update on Sunday’s service disruption

#171

Post by mikro2nd » Tue, Jun 04, 2019, 1:38 PM UTC

grogers wrote: post #168

One of my favorite patterns for updating configuration in-band I learned from Juniper routers. When you enact a new configuration you can have it automatically rollack after some period of time unless you confirm the configuration. Often the pattern is to intentionally have it roll back after a short period (e.g. one minute), then again after a longer period, (e.g. 10 minutes) and the on the last time you make it per…

I recall one occasion when I thanked all the gods for having been taught this pattern... firewall changes made from 2000km away that went pear-shaped...

Re: An update on Sunday’s service disruption

#172

Post by ldng » Tue, Jun 04, 2019, 1:39 PM UTC

gamegod wrote: post #167

markphip wrote: post #149

Is it just me or is this lacking any acknowledgment of the impact it had on GCE and all of the third parties that were impacted by this. They make it sound like a few people could not watch YouTube videos and even fewer people had some email disruption but this outage had a lot more impact than that. As just one example, a huge number of Shopify sites were impacted by this as were I am sure a number of other SaaS bus…

The network became congested, and our networking systems correctly triaged the traffic overload and dropped larger, less latency-sensitive traffic in order to preserve smaller latency-sensitive traffic flows... Overall, YouTube measured a 10% drop in global views during the incident... So what I'm hearing is that while Google Cloud Pub/Sub was down for hours, crippling my SaaS business, Google was prioritizing traffi…

Cat videos would be generating more revenue?

Re: An update on Sunday’s service disruption

#173

Post by dabei » Tue, Jun 04, 2019, 1:41 PM UTC

markphip wrote: post #149

Is it just me or is this lacking any acknowledgment of the impact it had on GCE and all of the third parties that were impacted by this. They make it sound like a few people could not watch YouTube videos and even fewer people had some email disruption but this outage had a lot more impact than that. As just one example, a huge number of Shopify sites were impacted by this as were I am sure a number of other SaaS bus…

Absolutely. I guess they want to avoid drawing attention to the full impact on businesses built on Google Cloud, before having a concerted effort across PR, Legal, Eng.

Re: An update on Sunday’s service disruption

#174

Post by enneff » Tue, Jun 04, 2019, 1:50 PM UTC

gamegod wrote: post #167

markphip wrote: post #149

Is it just me or is this lacking any acknowledgment of the impact it had on GCE and all of the third parties that were impacted by this. They make it sound like a few people could not watch YouTube videos and even fewer people had some email disruption but this outage had a lot more impact than that. As just one example, a huge number of Shopify sites were impacted by this as were I am sure a number of other SaaS bus…

The network became congested, and our networking systems correctly triaged the traffic overload and dropped larger, less latency-sensitive traffic in order to preserve smaller latency-sensitive traffic flows... Overall, YouTube measured a 10% drop in global views during the incident... So what I'm hearing is that while Google Cloud Pub/Sub was down for hours, crippling my SaaS business, Google was prioritizing traffi…

10% of YouTube views is a significant portion of ALL internet traffic. They weren’t prioritising YouTube, they used that as an example to describe the magnitude of the outage.

It is easy to measure YouTube views, somewhat harder to measure the effect on a service as complex as GCP. I am sure they’ll have more to say about the effect on GCP services once they have more detailed analysis.

Re: An update on Sunday’s service disruption

#175

Post by QuercusMax » Tue, Jun 04, 2019, 1:50 PM UTC

joncrane wrote: post #157

jrockway wrote: post #143

This answered all the questions I had. I was really racking my brain on what one system at Google could go down to cause this much damage, but it makes perfect sense that bandwidth becoming unavailable and everything in the "default" or "bulk" traffic class being dropped would do it. The real question is whether the fix will be to not reduce bandwidth accidentally, or to upgrade customer traffic to a higher QoS class…

Question since you're an ex-Googler: is part of the scheme is to segment the traffic by revenue generation. I bet that paying customers get the priority, and free services get de-prioritized. Is there any other analysis as well? For example, among the free services, maybe they rank them based on how much people will notice/how much press it would get if that service slowed down or stopped?

Nothing like that as far as I know. There are several tiers of Assured Forwarding, and after that it's Best Effort.

Re: An update on Sunday’s service disruption

#176

Post by jacobsenscott » Tue, Jun 04, 2019, 1:50 PM UTC

JMTQp8lwXL wrote: post #64

theevilsharpie wrote: post #58

Earlier quoted context omitted.

Google kept their status page up to date as the outage was progressing, and now (the day after the outage), they've provided an apology and a preliminary explanation of what happened. If that's not sufficient, what more are you looking for, and what other large cloud providers consistently meet that standard?

The status page wasn't up to date, but I don't have any way of backing up that claim. It's certainly isn't tied to any automated system failure reporting -- the status page seems to require manual updates. When minutes turn into hours, and no updates are available on the status page, it certainly doesn't leave me at ease.

Gmail was down for me for at least an hour and all the lights were green on their status page

Re: An update on Sunday’s service disruption

#177

Post by fragmede » Tue, Jun 04, 2019, 1:52 PM UTC

jrockway wrote: post #143

This answered all the questions I had. I was really racking my brain on what one system at Google could go down to cause this much damage, but it makes perfect sense that bandwidth becoming unavailable and everything in the "default" or "bulk" traffic class being dropped would do it. The real question is whether the fix will be to not reduce bandwidth accidentally, or to upgrade customer traffic to a higher QoS class…

EBS bandwidth limits are still quite visible under certain operations, simply due to the nature of the product, and anyone building large systems on top of it should understand those limitations, even if not directly visible, due to the nature of EBS (block attached storage connected to virtual machines of semi-unknown location), compared to S3 (generic blob store).

(An NDA with AWS also helps.)

Re: An update on Sunday’s service disruption

#178

Post by RKearney » Tue, Jun 04, 2019, 1:56 PM UTC

This sounds very much like what caused Amazon’s last S3 outage. A configuration change applied to more servers than expected. It’s unfortunate to see Google didn’t take action to prevent this after it happened to AWS and instead waited until it happened to them before realizing they need to put in safeguards against this.

Looking forward to the final write up on this with more details, but at first glance the cause looks just like S3’s last outage.

Re: An update on Sunday’s service disruption

#179

Post by xtracto » Tue, Jun 04, 2019, 1:57 PM UTC

grogers wrote: post #168

One of my favorite patterns for updating configuration in-band I learned from Juniper routers. When you enact a new configuration you can have it automatically rollack after some period of time unless you confirm the configuration. Often the pattern is to intentionally have it roll back after a short period (e.g. one minute), then again after a longer period, (e.g. 10 minutes) and the on the last time you make it per…

This has been in Windows in the screen resolution configuration option for some time.

Re: An update on Sunday’s service disruption

#180

Post by fragmede » Tue, Jun 04, 2019, 1:58 PM UTC

grogers wrote: post #168

One of my favorite patterns for updating configuration in-band I learned from Juniper routers. When you enact a new configuration you can have it automatically rollack after some period of time unless you confirm the configuration. Often the pattern is to intentionally have it roll back after a short period (e.g. one minute), then again after a longer period, (e.g. 10 minutes) and the on the last time you make it per…

This is called a watchdog timer, and is frequently used in embedded systems where "bare-metal" "OS" upgrades need to occur. You think it's stressful upgrading a router on the other side of the planet, try upgrading the firmware on the Mars Rover!

(I've only heard stories.)