Earlier quoted context omitted.
This is called a watchdog timer, and is frequently used in embedded systems where "bare-metal" "OS" upgrades need to occur. You think it's stressful upgrading a router on the other side of the planet, try upgrading the firmware on the Mars Rover! (I've only heard stories.)
That's not really what a whatchdog timer is though. https://en.m.wikipedia.org/wiki/Watchdog_timer
An update on Sunday’s service disruption
191–200 of 220 posts
Re: An update on Sunday’s service disruption
#192One of my favorite patterns for updating configuration in-band I learned from Juniper routers. When you enact a new configuration you can have it automatically rollack after some period of time unless you confirm the configuration. Often the pattern is to intentionally have it roll back after a short period (e.g. one minute), then again after a longer period, (e.g. 10 minutes) and the on the last time you make it per…
This has been in Windows in the screen resolution configuration option for some time.
Re: An update on Sunday’s service disruption
#193Earlier quoted context omitted.
TBH the linked update is a pretty accurate summary of the postmortem in my opinion.
+1 (I'm a Googler, opinions my own) As someone who has been on oncall for 3 year and done a decent amount of production support, this public doc better explains the cause than the internal one if you aren't well versed in the underlying infra. A config change reduced used network capacity by half, and then things started falling over. And pushing the fix took a while due to the now overloaded network.
Re: An update on Sunday’s service disruption
#194One of my favorite patterns for updating configuration in-band I learned from Juniper routers. When you enact a new configuration you can have it automatically rollack after some period of time unless you confirm the configuration. Often the pattern is to intentionally have it roll back after a short period (e.g. one minute), then again after a longer period, (e.g. 10 minutes) and the on the last time you make it per…
Yes|No
Re: An update on Sunday’s service disruption
#195This is a surprisingly vague postmortem. No timeline, no specific identification of affected regions. And no explanation of why a configuration change that was (apparently) made with a single command required so much effort to undo, or why repair efforts were hampered when (again, apparently) the network was successfully prioritizing high-priority traffic. Even for a public postmortem, this seems pretty weak.
As many have pointed out, this was not actually the postmortem, just an "update". I still find it pretty weak. There has been plenty of time to assemble basic details, such as a rough timeline. Yes they are busy with the actual response and cleanup, but this is a big professional team. The update feels less like "we're on it, here's what we have so far, more to come", and more like a bland PR minimization exercise.
I'll note that they've found time to determine that "For most Google users there was little or no visible change to their services", "YouTube measured a 10% drop in global views during the incident", and "approximately 1% of active Gmail users had problems", and yet they don't mention the time at which the incident was fully resolved or even when it started! Did the impact last for a few minutes, an hour, or multiple hours? Reading this, I have no idea. But I do learn that "the Google teams were keenly aware that every minute which passed represented another minute of user impact", they "brought on additional help to parallelize restoration efforts", "networking systems correctly triaged the traffic overload", and "low-bandwidth services like Google Search recorded only a short-lived increase in latency". In other words, this update on what seems to have been a very major incident consists mostly of vague, positive statements. It does say "we take it very seriously", but it doesn't make me believe it.
(I would add that I worked at Google from 2006 to 2010, and based on that experience I'm sure they are taking this very seriously and that there will be an excellent internal postmortem. But man, reading this sure makes it hard to remember that.)
Re: An update on Sunday’s service disruption
#196Is it just me or is this lacking any acknowledgment of the impact it had on GCE and all of the third parties that were impacted by this. They make it sound like a few people could not watch YouTube videos and even fewer people had some email disruption but this outage had a lot more impact than that. As just one example, a huge number of Shopify sites were impacted by this as were I am sure a number of other SaaS bus…
Re: An update on Sunday’s service disruption
#197This answered all the questions I had. I was really racking my brain on what one system at Google could go down to cause this much damage, but it makes perfect sense that bandwidth becoming unavailable and everything in the "default" or "bulk" traffic class being dropped would do it. The real question is whether the fix will be to not reduce bandwidth accidentally, or to upgrade customer traffic to a higher QoS class…
>Engineers building apps that depend on that know the limitations. As someone who works at a slightly smaller tech company with of similar age with similar infrastructure I assure you this is not the case. Engineers are building things that rely on other things that rely on other things. There's a point where people don't know what their dependencies are. I wouldn't be surprised if nobody actually knew there was cust…
That's what a distributed system is: a system in which you can't get your work done because a system you've never heard of has failed. (I had that attributed to Butler Lampson, but searching turns up Leslie Lamport instead)
Re: An update on Sunday’s service disruption
#198Earlier quoted context omitted.
The network became congested, and our networking systems correctly triaged the traffic overload and dropped larger, less latency-sensitive traffic in order to preserve smaller latency-sensitive traffic flows... Overall, YouTube measured a 10% drop in global views during the incident... So what I'm hearing is that while Google Cloud Pub/Sub was down for hours, crippling my SaaS business, Google was prioritizing traffi…
Considering this was seemingly a mostly North America-affecting networking issue, and the 10% reduction in views was global, it doesn't sound like YouTube got much of a priority - quite a lot of the videos that were actually view-able in affected regions during the outage may simply have been served from edge caches. Disclaimer: no inside knowledge, the above is pure supposition
It's undermines their GCP business in a big way too - It makes you think that if they had to choose, they would throw their GCP customers under the bus to preserve their own other services. The value proposition of GCP is greatly diminished then in comparison to a dedicated cloud provider like DigitalOcean, who has no other competing interests. This changed the way I view some of these cloud providers.
eg. If Google had to prioritize ad network traffic over GCP, there's no question the ad network would get priority. But why not just go with a different provider who doesn't have to make that compromise?
Re: An update on Sunday’s service disruption
#199Earlier quoted context omitted.
This has been in Windows in the screen resolution configuration option for some time.
Ah, the lost joy of trying out various screen resolutions and refresh rates on your new monitor.