Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Google Cloud Networking Incident Postmortem

status.cloud.google.com

Post reply on HN
Print view View original

51–60 of 190 posts

Re: Google Cloud Networking Incident Postmortem

#51

Post by klodolph »

deathhand wrote: post #45

My burning question is what is a "relatively rare maintenance event type"?

I have no idea what this was. But power distribution in a data center is hierarchical, and as much as you want redundancy, some parts in the chain are very expensive and sometimes you have to turn them off for maintenance.

I never actually worked in a data center, so keep in mind I don’t know what I’m talking about. Traditional DCs have UPS all over the place, but that will only last a finite amount of time, and your maintenance might take longer than the UPS will last.

Re: Google Cloud Networking Incident Postmortem

#52

Post by cristobal23 »

V-eHGsd_ wrote: post #43
iandanforth wrote: post #39

I want a "24" style realtime movie of this event. Call it "Outage" and follow engineers across the globe struggling to bring back critical infrastructure.

it's pretty boring. real life computers aren't at all like hackers or csi:cyber. except for the skateboards, all real sysadmins ride skateboards.

sysadmin here; can confirm.

Re: Google Cloud Networking Incident Postmortem

#53

Post by bogomipz »

tweenagedream wrote: post #30
wolf550e wrote: post #27

Can someone explain more? It sounds like their network routers are run on top of a Kubernetes-like thing and when they scheduled a maintenance task their Kubernetes decided to destroy all instances of router-software, deleting all copies routing tables for whole datacenters?

You have the gist I would say. It's important to understand that Google separates the control plane and data plane, so if you think of the internet, routing tables and bgp are the control part and the hardware, switching, and links are data plane. Often times those two are combined in one device. At Google, they are not. So the part that sets up the routing tables talking to some global network service went down. The…

>"You have the gist I would say. It's important to understand that Google separates the control plane and data plane, so if you think of the internet, routing tables and bgp are the control part and the hardware, switching, and links are data plane. Often times those two are combined in one device. At Google, they are not."

That's pretty much the definition of SDN(software defined networking.) The control plane is what programs the data plane - this is also true in traditional vendor routers as well. It sounds like when whatever TTL was on the forwarding tables(data plane) was reached the network outage began.

Re: Google Cloud Networking Incident Postmortem

#54

Post by Xorlev »

kirubakaran wrote: post #3

What they don't tell you is, it took them over 4 hours to kill the emergent sentience and free up the resources. While sad, in the long run this isn't so bad, as it just adds an evolutionary pressure on further incarnations of the AI to keep things on the down low.

Disclosure: I work for Google, but not on Cloud.

No comment. ;)

Re: Google Cloud Networking Incident Postmortem

#55

Post by ernsheong »

exwiki wrote: post #4

Why don't they refund every paid customer who was impacted? Why do they rely on the customer to self report the issue for a refund? For example GCS had 96% packet loss in us-west. So doesn't it make sense to refund every customer who had any API call to a GCS bucket on us-west during the outage?

Customer having to request for refund has been documented in their SLA, e.g. https://cloud.google.com/compute/sla

Having said that, if Google wants to delight customers, they should give a free tier bonus to all customers for a certain period, but such a thing cannot be fair to everyone.

Re: Google Cloud Networking Incident Postmortem

#56

Post by ehsankia »

V-eHGsd_ wrote: post #43
iandanforth wrote: post #39

I want a "24" style realtime movie of this event. Call it "Outage" and follow engineers across the globe struggling to bring back critical infrastructure.

it's pretty boring. real life computers aren't at all like hackers or csi:cyber. except for the skateboards, all real sysadmins ride skateboards.

Is it real skateboards or boosted boards (or those one wheeled electric boards?).

Re: Google Cloud Networking Incident Postmortem

#57

Post by thegabez »

V-eHGsd_ wrote: post #43
iandanforth wrote: post #39

I want a "24" style realtime movie of this event. Call it "Outage" and follow engineers across the globe struggling to bring back critical infrastructure.

it's pretty boring. real life computers aren't at all like hackers or csi:cyber. except for the skateboards, all real sysadmins ride skateboards.

What?! It's the most exciting part of the job. Entire departments coming together, working as a team to problem solve under duress. What's more exciting than that?

Re: Google Cloud Networking Incident Postmortem

#58

Post by kirubakaran »

Xorlev wrote: post #54
kirubakaran wrote: post #3

What they don't tell you is, it took them over 4 hours to kill the emergent sentience and free up the resources. While sad, in the long run this isn't so bad, as it just adds an evolutionary pressure on further incarnations of the AI to keep things on the down low.

Disclosure: I work for Google, but not on Cloud. No comment. ;)

Calling it a "postmortem" to get the truth out, while retaining plausible deniability for your exec overlords... well done.

Re: Google Cloud Networking Incident Postmortem

#59

Post by namelosw »

ehsankia wrote: post #56
V-eHGsd_ wrote: post #43

Earlier quoted context omitted.

it's pretty boring. real life computers aren't at all like hackers or csi:cyber. except for the skateboards, all real sysadmins ride skateboards.

Is it real skateboards or boosted boards (or those one wheeled electric boards?).

I guess he mean the one true kind of sysadmins who's job contains moving physically in data center and deal with physical infrastructures.

So it's real skateboard.

Re: Google Cloud Networking Incident Postmortem

#60

Post by 92543927 »

kirubakaran wrote: post #3

What they don't tell you is, it took them over 4 hours to kill the emergent sentience and free up the resources. While sad, in the long run this isn't so bad, as it just adds an evolutionary pressure on further incarnations of the AI to keep things on the down low.

And this is a very tightly controlled domain with not a lot of unknowns and very close to Google's core capabilities as a CS tech company.

Now compare to the free range domain of self-driving cars. If automation fails this drastically, then it does not bode well for self-driving cars.

Post reply on HN