201–210 of 220 posts
Earlier quoted context omitted.
Ah, the lost joy of trying out various screen resolutions and refresh rates on your new monitor.
nvidia drivers (probably ati too) let you create custom resolutions. I got my monitor to do 1080p@120hz that way. (default supported is only 60hz)
Earlier quoted context omitted.
That's not really what a whatchdog timer is though. https://en.m.wikipedia.org/wiki/Watchdog_timer
While Watchdog Timer is more so an external-to-the-cpu system for embedded systems, I'm not sure if there's a word for it in the software world other then some form of timeout?
> In essence, the root cause of Sunday’s disruption was a configuration change I feel like I hear about config changes breaking these cloud hosts so often it might as well be a meme. Is there a reason why it's usually configurations to blame vs code, hardware, etc?
Why is it very hard? Because when you are the size of Google, there is no second version of prod to test things in, so the usual software engineering solution of trying the new thing in isolation and checking if it worked is unrealistic.
I think these constant failures from config changes should cause folks to re-evaluate how they do config changes though. If we can't just do a green/blue deploy of config changes like this, we probably need some other solution, whether it be the watchdog timers mentioned elsewhere in the thread, or some system that is able to show you the impact of a config change before it takes effect (probably more realistic for single services such as networking, rather than all config changes).
Earlier quoted context omitted.
> I don't think it's helpful to treat these as isolated one-offs caused by a bogus configuration. That’s why I said the config change is “just the trigger”. Root cause analysis will generally result in multiple causes for any problem. > Perhaps what is required is a completely different attitude to config changes, which treats them as testable, applies them incrementally and allows simple rollback. Google already has…
Unfortunately, in the real world config changes are hard to test. I'm not trying to suggest that I know what the answer is and it's simple, just that config does need more work, it now seems to be the point of failure for all these big networks (rather than hardware or code changes). These big providers seem to have almost entirely tackled hardware changes and software changes as causes of outages, and configs have b…
Think of it as input, to a global network of inter-dependent distributed decentralized programs, which control other programs, that then change inputs, that change the programs again, and which are never "off", but always just shifting where bits are.
Imagine a cloud-based web application. You've got your app code, and let's say an embedded HTTP server. The code needs to run somewhere, on Lambda, or ECS, or EC2. You need an S3 bucket, a load balancer, an internet gateway, security groups, Route53 records, roles, policies, VPCs. Each of those has a config, and when any is applied it affects all the other components, because they're part of a chain of dependencies. Now make the changes in multiple regions. Tests add up quickly, and that's just in ways that were obvious. Now add tests for outages of each component, timeouts, bad data, resource starvation, etc. Just a simple web service can mean tens of thousands of tests.
We imagine that because the things we're manipulating are digital, they must behave predictably. But they don't. Look at all the databases tested by Jepsen[1]. People who are intelligent and are paid lots of money still regularly create distributed systems with huge flaws that affect production systems. Creating a complex, predictable system is h a r d (and for Turing-complete systems, actually impossible - see the halting problem).
Earlier quoted context omitted.
Considering this was seemingly a mostly North America-affecting networking issue, and the 10% reduction in views was global, it doesn't sound like YouTube got much of a priority - quite a lot of the videos that were actually view-able in affected regions during the outage may simply have been served from edge caches. Disclaimer: no inside knowledge, the above is pure supposition
You're right, but I wanted to point out their poor wording. They shouldn't downplay the huge impact on GCP customers in one paragraph, and then gloat about YouTube being fine in the next. It's undermines their GCP business in a big way too - It makes you think that if they had to choose, they would throw their GCP customers under the bus to preserve their own other services. The value proposition of GCP is greatly di…
Earlier quoted context omitted.
While Watchdog Timer is more so an external-to-the-cpu system for embedded systems, I'm not sure if there's a word for it in the software world other then some form of timeout?
A watchdog timer is so named because it watches for some undesirable condition (usually a hang) and takes action if the condition lasts too long. This sort of auto-rollback is not that because it doesn't detect if something is bad, just rolls it back unless you tell it not to.
> A watchdog timer is an electronic timer that is used to detect and recover from computer malfunctions. During normal operation, the computer regularly resets the watchdog timer to prevent it from elapsing, or "timing out".
It doesn't care about the state of anything except it's timer, and the only way to prevent it from activating is to reset the timer or disable the watchdog altogether.
That would still make sense in terms of auto-roll backs. You can't trust the state as a miss-configuration makes it unreliable.
The only difference I see from "auto-rollbacks" and "watchdog timers" is that watchdog timers are usually meant to be permanent, while auto-rollbacks are temporary (once you confirm it the auto-rollback never occurs again).
Earlier quoted context omitted.
Unfortunately, in the real world config changes are hard to test. I'm not trying to suggest that I know what the answer is and it's simple, just that config does need more work, it now seems to be the point of failure for all these big networks (rather than hardware or code changes). These big providers seem to have almost entirely tackled hardware changes and software changes as causes of outages, and configs have b…
Don't think of it as "config". Think of it as input, to a global network of inter-dependent distributed decentralized programs, which control other programs, that then change inputs, that change the programs again, and which are never "off", but always just shifting where bits are. Imagine a cloud-based web application. You've got your app code, and let's say an embedded HTTP server. The code needs to run somewhere,…
There are other ways to control change and limit breakage other than just tests - dev networks at smaller scale, canaries, truly segregated networks, truly separate control networks for these inputs etc. All have downsides but there are lots of options.
We would not accept a program that rewrites itself in response to myriad inputs and is therefore highly unpredictable and unreliable, and config/infrastructure should be held to the same standard.
Earlier quoted context omitted.
A watchdog timer is so named because it watches for some undesirable condition (usually a hang) and takes action if the condition lasts too long. This sort of auto-rollback is not that because it doesn't detect if something is bad, just rolls it back unless you tell it not to.
Actually I think a watchdog timer doesn't watch anything except it's timer. > A watchdog timer is an electronic timer that is used to detect and recover from computer malfunctions. During normal operation, the computer regularly resets the watchdog timer to prevent it from elapsing, or "timing out". It doesn't care about the state of anything except it's timer, and the only way to prevent it from activating is to res…
I guess I was thinking of the periodic timer reset as part of the watchdog mechanism. Maybe another difference is whether the interaction with the timer is manual.
Earlier quoted context omitted.
While Watchdog Timer is more so an external-to-the-cpu system for embedded systems, I'm not sure if there's a word for it in the software world other then some form of timeout?
A watchdog timer is so named because it watches for some undesirable condition (usually a hang) and takes action if the condition lasts too long. This sort of auto-rollback is not that because it doesn't detect if something is bad, just rolls it back unless you tell it not to.
This sounds similar to what was being described.
Is it just me or is this lacking any acknowledgment of the impact it had on GCE and all of the third parties that were impacted by this. They make it sound like a few people could not watch YouTube videos and even fewer people had some email disruption but this outage had a lot more impact than that. As just one example, a huge number of Shopify sites were impacted by this as were I am sure a number of other SaaS bus…
The network became congested, and our networking systems correctly triaged the traffic overload and dropped larger, less latency-sensitive traffic in order to preserve smaller latency-sensitive traffic flows... Overall, YouTube measured a 10% drop in global views during the incident... So what I'm hearing is that while Google Cloud Pub/Sub was down for hours, crippling my SaaS business, Google was prioritizing traffi…
It was not. Youtube was unavailable to me, but gmail worked sporadically.