111–120 of 220 posts
This reminds me of the time I wanted to test packet loss for a VoIP app and used `tc` to introduce 95% packet loss on the office gateway and because of the packet loss I could not ssh into the box to turn it off... on Google scale.
How did you end up resolving your issue?
Earlier quoted context omitted.
> The description is vague about what devices ("servers") were misconfigured. "servers" when said by Googlers usually means processes that serve requests, not machines. Hopefully a future postmortem will provide more details. > How would restricted bandwidth utilization on servers cause network congestion... This is a common problem with load balancing if you ever use non-trivial configuration. Imagine you split 100…
> During normal operations, you usually want administrative traffic (e.g. config or executable updates) to be low-priority so it doesn't disrupt production traffic Honest question: is it not best practice to have an isolated, dedicated management network? I can’t for the life of me understand why a misconfig on the production network should hamper access through the admin network. Unless on Google’s scale it’s not th…
Earlier quoted context omitted.
> The description is vague about what devices ("servers") were misconfigured. "servers" when said by Googlers usually means processes that serve requests, not machines. Hopefully a future postmortem will provide more details. > How would restricted bandwidth utilization on servers cause network congestion... This is a common problem with load balancing if you ever use non-trivial configuration. Imagine you split 100…
> During normal operations, you usually want administrative traffic (e.g. config or executable updates) to be low-priority so it doesn't disrupt production traffic Honest question: is it not best practice to have an isolated, dedicated management network? I can’t for the life of me understand why a misconfig on the production network should hamper access through the admin network. Unless on Google’s scale it’s not th…
Earlier quoted context omitted.
> I don't think it's helpful to treat these as isolated one-offs caused by a bogus configuration. That’s why I said the config change is “just the trigger”. Root cause analysis will generally result in multiple causes for any problem. > Perhaps what is required is a completely different attitude to config changes, which treats them as testable, applies them incrementally and allows simple rollback. Google already has…
Unfortunately, in the real world config changes are hard to test. I'm not trying to suggest that I know what the answer is and it's simple, just that config does need more work, it now seems to be the point of failure for all these big networks (rather than hardware or code changes). These big providers seem to have almost entirely tackled hardware changes and software changes as causes of outages, and configs have b…
It’s easy to say things like “should never be global” and “should not be hard to test”. These are goals, and meanwhile the business must go on, you also have other goals, and you cannot spend your entire budget preventing network outages and testing configs.
The things you are suggesting—separate control plane, non-global configs, making them easy to test—you can find these suggestions in any book on operations. So forgive me if your comment makes me a bit angry.
Earlier quoted context omitted.
Fair, but I have friends on the west coast who were similarly impacted, while I am in the South. So it seems inaccurate to say it impacted a small number of users. Maybe compared to world wide, but I would imagine the US is the biggest market.
> However, for users who rely on services homed in the affected regions, the impact was substantial, particularly for services like YouTube or Google Cloud Storage which use large amounts of network bandwidth to operate. The post admits that. It clearly says that the impact on users in affected regions was significant but that some regions were barely affected. It would've been nice if they mentioned what regions. Bu…
" gcloud tells me:
WARNING: The following zones did not respond: us-west2, us-west2-a, southamerica-east1-c, us-west2-b, southamerica-east1, us-east4-b, us-east4, us-east4-a, northamerica-northeast1-c, northamerica-northeast1-b, us-west2-c, southamerica-east1-b, northamerica-northeast1, southamerica-east1-a, northamerica-northeast1-a, us-east4-c. List results may be incomplete.
Luckily for us eu-west1 seems to be working normally."
So the user's outside these zones may have been unaffected, but if this is accurate it is a large number of users affected
Earlier quoted context omitted.
Google kept their status page up to date as the outage was progressing, and now (the day after the outage), they've provided an apology and a preliminary explanation of what happened. If that's not sufficient, what more are you looking for, and what other large cloud providers consistently meet that standard?
AWS's post outrage summaries are pretty much the gold standard. e.g. https://aws.amazon.com/message/2329B7/ https://aws.amazon.com/message/41926/ To be fair to Google, they haven't had enough time to perform a detailed autopsy, and some GCP incident summaries have shown meat on the bones e.g. https://status.cloud.google.com/incident/compute/16007 . And balancing the scales, the AWS status page is notorious for showin…
Every time I read a Google post-mortem, they seem to hand wave everything away as "a configuration error", "bug", or "bad deploy" and their resolution always has the generic "implement changes to things" that says absolutely nothing. Honestly, when the the causes of these massive disruptions are so simply dismissed, it portrays their system as frail amateur work.
Earlier quoted context omitted.
Google kept their status page up to date as the outage was progressing, and now (the day after the outage), they've provided an apology and a preliminary explanation of what happened. If that's not sufficient, what more are you looking for, and what other large cloud providers consistently meet that standard?
The status page wasn't up to date, but I don't have any way of backing up that claim. It's certainly isn't tied to any automated system failure reporting -- the status page seems to require manual updates. When minutes turn into hours, and no updates are available on the status page, it certainly doesn't leave me at ease.