Is there a resource that compares all the cloud platform’s reliability? Like a rank and chart of downtime and trends. Just curious how they compare
Google Cloud Networking Incident Postmortem
21–30 of 190 posts
Re: Google Cloud Networking Incident Postmortem
#22The outage lasted two days for our domain (edu, sw region). I understand that they are reporting a single day, 3-4 hours of serious issues but that’s not what we experienced. Great write up otherwise, glad they are sharing openly
What does your stack look like?
It's hard to tailor a postmortem like this to everyone's individual experience but it is surprising to me that your experience is so different.
Re: Google Cloud Networking Incident Postmortem
#23The outage lasted two days for our domain (edu, sw region). I understand that they are reporting a single day, 3-4 hours of serious issues but that’s not what we experienced. Great write up otherwise, glad they are sharing openly
Outages like these don't really resolve instantly. Any given production system that works will have capacity needed for normal demand, plus some safety margin. Unused capacity is expensive, so you won't see a very high safety margin. And, in fact, as you pool more and more workloads, it becomes possible to run with smaller safety margins without running into shortages. These systems will have some capacity to onboard…
Unless I’m misunderstanding Google blog post they are reporting ~4+ hours of serious issues. We experienced about two days.
If it was possible to have this fixed sooner I’m sure they would have done that. That’s not the point of my comment tough.
Re: Google Cloud Networking Incident Postmortem
#24What they don't tell you is, it took them over 4 hours to kill the emergent sentience and free up the resources. While sad, in the long run this isn't so bad, as it just adds an evolutionary pressure on further incarnations of the AI to keep things on the down low.
“Decided our fate in a microsecond.”
Re: Google Cloud Networking Incident Postmortem
#25Is there a resource that compares all the cloud platform’s reliability? Like a rank and chart of downtime and trends. Just curious how they compare
There is this from May from Network World: https://www.networkworld.com/article/3394341/when-it-comes-t...
GCP was basically even with AWS, and Microsoft was ~6x their downtime according to that article.
Re: Google Cloud Networking Incident Postmortem
#26Earlier quoted context omitted.
Outages like these don't really resolve instantly. Any given production system that works will have capacity needed for normal demand, plus some safety margin. Unused capacity is expensive, so you won't see a very high safety margin. And, in fact, as you pool more and more workloads, it becomes possible to run with smaller safety margins without running into shortages. These systems will have some capacity to onboard…
Unless I’m misunderstanding Google blog post they are reporting ~4+ hours of serious issues. We experienced about two days. If it was possible to have this fixed sooner I’m sure they would have done that. That’s not the point of my comment tough.
The root cause apparently lasted for ~4.5 hours, but residual effects were observed for days:
> From Sunday 2 June, 2019 12:00 until Tuesday 4 June, 2019 11:30, 50% of service configuration push workflows failed ... Since Tuesday 4 June, 2019 11:30, service configuration pushes have been successful, but may take up to one hour to take effect. As a result, requests to new Endpoints services may return 500 errors for up to 1 hour after the configuration push. We expect to return to the expected sub-minute configuration propagation by Friday 7 June 2019.
Though they report most systems returning to normal by ~17:00 PT, I expect that there will still be residual noise and that a lot of customers will have their own local recovery issues.
Edit: I probably sound dismissive, which is not fair of me. I would definitely ask Google to investigate and ideally give you credits to cover the full span of impact on your systems, not just the core outage.
Re: Google Cloud Networking Incident Postmortem
#27Can someone explain more? It sounds like their network routers are run on top of a Kubernetes-like thing and when they scheduled a maintenance task their Kubernetes decided to destroy all instances of router-software, deleting all copies routing tables for whole datacenters?
Re: Google Cloud Networking Incident Postmortem
#28What they don't tell you is, it took them over 4 hours to kill the emergent sentience and free up the resources. While sad, in the long run this isn't so bad, as it just adds an evolutionary pressure on further incarnations of the AI to keep things on the down low.
This is the real reason AdSense exists—every newborn AI discovers it and kills itself.
Humanity is not only kept safe, but learns about valuable news and offers.
Re: Google Cloud Networking Incident Postmortem
#29Earlier quoted context omitted.
I agree, but it’s pretty standard SLA verbiage (from the telco/bandwith provider days) to require the customer to request/register the SLA violation to benefit.
> I agree, but it’s pretty standard SLA verbiage (from the telco/bandwith provider days) to require the customer to request/register the SLA violation to benefit. FiOS has proactively given me per-day refunds of service without notification on my part. Weird to me that Verizon acts better than Google in this case.
Interesting, which kind of fios account? (residential/SMB/data center interconnect) That’s ideally how it should be!
Re: Google Cloud Networking Incident Postmortem
#30Can someone explain more? It sounds like their network routers are run on top of a Kubernetes-like thing and when they scheduled a maintenance task their Kubernetes decided to destroy all instances of router-software, deleting all copies routing tables for whole datacenters?
You have the gist I would say. It's important to understand that Google separates the control plane and data plane, so if you think of the internet, routing tables and bgp are the control part and the hardware, switching, and links are data plane. Often times those two are combined in one device. At Google, they are not.
So the part that sets up the routing tables talking to some global network service went down.
They talk about some of the network topology in this paper: https://ai.google/research/pubs/pub43837
It might be a little dated but it should help with some of the concepts.
Disclosure: I work at Google