Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Google Cloud Networking Incident Postmortem

status.cloud.google.com

Post reply on HN
Print view View original

71–80 of 190 posts

Re: Google Cloud Networking Incident Postmortem

#71

Post by stanley »

jasonjei wrote: post #47
exwiki wrote: post #4

Why don't they refund every paid customer who was impacted? Why do they rely on the customer to self report the issue for a refund? For example GCS had 96% packet loss in us-west. So doesn't it make sense to refund every customer who had any API call to a GCS bucket on us-west during the outage?

My question is if you had to pay for Google AdWords and your site was inaccessible due to GCloud outage, do you have recourse on SLA for paid clicks? Or is that money paid to Google AdWords lost?

In the same boat.

The icing on the cake was they disapproved one of the ads due to the destination URL not loading.. which was in itself surprising, because everything outside of the affected region was running fine.

Re: Google Cloud Networking Incident Postmortem

#72

Post by anonfunction »

The only way to get SLA credits is requesting it. This is very disappointing.

  SLA CREDITS
  
  If you believe your paid application experienced an SLA violation 
  as a result of this incident, please populate the SLA credit request:
  https://support.google.com/cloud/contact/cloud_platform_sla

Re: Google Cloud Networking Incident Postmortem

#73

Post by opportune »

roganartu wrote: post #68
ljoshua wrote: post #2

Having only ever seen one major outage event in person (at a financial institution that hadn't yet come up with an incident response plan; cue three days of madness), I would love to be a fly on the wall at Google or other well-established engineering orgs when something like this goes down. I'd love to see the red binders come down off the shelf, people organize into incident response groups, and watch as a root cau…

I used to be an SRE at Atlassian in Sydney on a team that regularly dealt with high-severity incidents, and I was an incident manager for probably 5-10 high severity Jira cloud incidents during my tenure too, so perhaps I can give some insight. I left because the SRE org in general at the time was too reactionary, but their incident response process was quite mature (perhaps by necessity). The first thing I'll say is…

Yes this is also how it's done at other large orgs. But one key to a quick response is for every low-level team to have at least one engineer on call at any given time. This makes it so any SRE team can engage with true "owners" of the offending code ASAP.

Also during an incident, fingers are never publicly/embarrassingly pointed nor are people blamed. It's all about identifying and fixing the issue as fast as possible, fixing it, and going back to sleep/work/home. For better or worse, incidents become routine so everyone knows exactly what do and that as long as the incident is resolved soon, it's not the end of the world, so no histrionics are required.

Re: Google Cloud Networking Incident Postmortem

#74

Post by dnautics »

carlsborg wrote: post #37

I was curious to know how cascading failures in one region effected other regions. Impact was " ...increased latency, intermittent errors, and connectivity loss to instances in us-central1, us-east1, us-east4, us-west2, northamerica-northeast1, and southamerica-east1." Answer, and the root cause summarized: Maintenance started in a physical location, and then "... the automation software created a list of jobs to des…

> Debugging the problem was significantly hampered by failure of tools competing over use of the now-congested network.

Man that's got to suck.

Re: Google Cloud Networking Incident Postmortem

#76

Post by illumin8 »

oceliker wrote: post #70
exwiki wrote: post #4

Why don't they refund every paid customer who was impacted? Why do they rely on the customer to self report the issue for a refund? For example GCS had 96% packet loss in us-west. So doesn't it make sense to refund every customer who had any API call to a GCS bucket on us-west during the outage?

Not directly GCS-related, but there was a big Youtube TV outage during the World Cup of last year (I think it was during semi-finals?). Google did apologize, but they only offered a free week of Youtube TV, which they implemented by charging me a week later than usual. I didn't feel compensated at all (it was a pretty important game that I missed!)

Wow, what a dick move...

Re: Google Cloud Networking Incident Postmortem

#78

Post by fouc »

anonfunction wrote: post #72

The only way to get SLA credits is requesting it. This is very disappointing. SLA CREDITS If you believe your paid application experienced an SLA violation as a result of this incident, please populate the SLA credit request: https://support.google.com/cloud/contact/cloud_platform_sla

That does seem questionable. They should be able to detect who was affected in the first place.

Re: Google Cloud Networking Incident Postmortem

#79

Post by illumin8 »

tweenagedream wrote: post #30
wolf550e wrote: post #27

Can someone explain more? It sounds like their network routers are run on top of a Kubernetes-like thing and when they scheduled a maintenance task their Kubernetes decided to destroy all instances of router-software, deleting all copies routing tables for whole datacenters?

You have the gist I would say. It's important to understand that Google separates the control plane and data plane, so if you think of the internet, routing tables and bgp are the control part and the hardware, switching, and links are data plane. Often times those two are combined in one device. At Google, they are not. So the part that sets up the routing tables talking to some global network service went down. The…

It shouldn't. Amazon believes in strict regional isolation, which means that outages only impact 1 region and not multiple. They also stagger their releases across regions to minimize the impact of any breaking changes (however unexptected...)

Re: Google Cloud Networking Incident Postmortem

#80

Post by tubaguy50035 »

scotchio wrote: post #21

Is there a resource that compares all the cloud platform’s reliability? Like a rank and chart of downtime and trends. Just curious how they compare

Slightly off topic rant follows: I don't see a lot of tech sites talk about the fact that Azure and GCP have multi-region outages. Everybody sees this kind of thing and goes "shrug, an outage". No, this is not okay. We have multiple regions for a reason. Making an application support multi-region is HARD and COSTLY. If I invest that into my app, I never want it go down due to a configuration push. There has never been an AWS incident across multiple regions (us-east-1, us-west-2, etc). That is a pretty big deal to me.

Whenever I post this somebody comes along and says "well that one time us-east-1 went down and everybody was using the generic S3 endpoints so it took everything down". This is true, and the ASG and EBS services in other regions apparently were. BUT, if you invested the time to ensure your application could be multi-region and you hosted on AWS, you would not have seen an outage. Scaling and snapshots might not have worked, but it would not have been the 96.2% packet drop that GCP is showing here and your end users likely would not have noticed.

The articles that track outages at the different cloud vendors really should be pushing this.

Post reply on HN