Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Google Cloud Networking Incident Postmortem

status.cloud.google.com

Post reply on HN
Print view View original

171–180 of 190 posts

Re: Google Cloud Networking Incident Postmortem

#171

Post by asark »

mslot wrote: post #149
carlsborg wrote: post #37

I was curious to know how cascading failures in one region effected other regions. Impact was " ...increased latency, intermittent errors, and connectivity loss to instances in us-central1, us-east1, us-east4, us-west2, northamerica-northeast1, and southamerica-east1." Answer, and the root cause summarized: Maintenance started in a physical location, and then "... the automation software created a list of jobs to des…

At some point people realized servers are prone to failure. They then started deploying their system redundantly to multiple servers in the data center (AZ) to increase availability. This helped, but created consistency issues. To fix this people started building multi-server software systems, creating dependencies across servers that weakened overall availability. At some point people realized multi-server systems w…

I honestly think a one- or two-server setup with scripted server re-creation, proven in Vagrant or whatever, and maybe using Docker but only to isolate services on those one or two servers and make it easier to re-create, tested frequently by spinning up local dev copies, and (obviously) with backups, is probably a stabler, cheaper, and higher-availability set-up for the vast majority of use cases. Even if predicted, scheduled downtime is somewhat higher, it's probably worth it for the many benefits.

But no, cloud everything.

Re: Google Cloud Networking Incident Postmortem

#172

Post by di4na »

ethbro wrote: post #162
di4na wrote: post #147

Earlier quoted context omitted.

It is not surprising at all. Mission Control was forged in the fire (literally for Apollo 1) and they are one of the most visible "incident team" we know about. I highly advise to read Gene Kranz memoirs "Failure is not an Option" if you work in that kind of environment.

I heard recently that he never said that. Apparently it was mentioned when the Apollo 13 script writers were gathering stories at NASA, they liked it, and then gave it to the Kranz character. Who then decided, "Hey, if everyone thinks I said it..." and titled his memoir.

Yep exactly.

Re: Google Cloud Networking Incident Postmortem

#173

Post by parliament32 »

exwiki wrote: post #4

Why don't they refund every paid customer who was impacted? Why do they rely on the customer to self report the issue for a refund? For example GCS had 96% packet loss in us-west. So doesn't it make sense to refund every customer who had any API call to a GCS bucket on us-west during the outage?

Same reason mail-in rebates used to be a huge thing on physical products.

They're well aware that ~50% won't bother, so that $10 discount per unit effectively becomes a $5 discount.

Re: Google Cloud Networking Incident Postmortem

#174

Post by dancek »

swebs wrote: post #142
hansflying wrote: post #89

Google has a huge quality problem and their service is extremely unreliable. Another 3-day-outage in kubernetes: https://news.ycombinator.com/item?id=18428497 login issues: https://news.ycombinator.com/item?id=19687029 storage system outage: https://news.ycombinator.com/item?id=19392452 ... So, basically Google created the most unreliable cloud system in the world.

>So, basically Google created the most unreliable cloud system in the world I'm pretty sure that title goes to Azure

You people probably haven't used IBM Cloud (or Bluemix, as it used to be). We inherited one application there, and boy was life stressful. There were already plans to move elsewhere, and then one day our managed production database was down. Took me something like ten hours to build a new production system elsewhere from backups, but it took longer for the engineers to fix the database.

Re: Google Cloud Networking Incident Postmortem

#175

Post by tetha »

Moru wrote: post #130
tetha wrote: post #98

Earlier quoted context omitted.

Someone at our place recently commented that the ops team during an incident strongly feels like NASA mission control in critical moments[1]. I wanted to protest, but that's surprisingly accurate. > And if anyone is screaming or gets emotional, they should not be in that room. If someone starts yelling around in my incident war room for no reason, they get thrown out. I'm a calm and quiet person, but bugging around d…

When the insident is over, does it look like 55:50 in that video? :-)

We recently had a 15 month long project almost fail due to some stupid shit and a really wonky error no one understood so far. as in "Keep the customer on the phone we have 3 possible things to try and don't hang up! No one leaves that call until I'm out of hacks to deploy!" That evening we had the entire ops team in the houston-mode for several hours.

And yes, once we had a workaround in place the customer accepted, we reacted like that. Except we also had our critical-incident whiskey go around. Then the CEO walked in to congratulate us on that project. Whoops. But he's a good sport, so good times. :)

Re: Google Cloud Networking Incident Postmortem

#176

Post by oceliker »

chipperyman573 wrote: post #170
oceliker wrote: post #70

Earlier quoted context omitted.

Not directly GCS-related, but there was a big Youtube TV outage during the World Cup of last year (I think it was during semi-finals?). Google did apologize, but they only offered a free week of Youtube TV, which they implemented by charging me a week later than usual. I didn't feel compensated at all (it was a pretty important game that I missed!)

What would you prefer they did instead?

Honestly, just the apology would be better. I felt I was being tricked. I end up paying each month anyway; it changed nothing for me.

Re: Google Cloud Networking Incident Postmortem

#177

Post by bnjms »

_n_b_ wrote: post #145
smileypete wrote: post #137

Earlier quoted context omitted.

Use a 28k modem? :)

An anecdote: my (not-IT) company does exactly this for out-of-band management... except, in one small satellite location, the phone company no longer provided any copper POTS lines; all they could do was an RJ-11 jack out of the ONT that was backhauled as (lossy) VoIP. So the modem couldn't be made to work. My point being, it seems that modems are becoming less-and-less viable for out-of-band management.

If this is a problem consider Opengear.

https://opengear.com/products/acm7000-resilience-gateway

Re: Google Cloud Networking Incident Postmortem

#178

Post by bnjms »

ddalex wrote: post #164
kbirkeland wrote: post #158

Earlier quoted context omitted.

A completely OOB management network is an amazingly high cost when you have presence all over the world. I don't think anybody has gone to the length to double up on dark fiber and OTN gear just for management traffic.

Hmm... With 5G each blade in the rack could get its own modem and sim card for OOB management.

Why would you do that when you could have 1 sim in a 96 port terminal server?

Re: Google Cloud Networking Incident Postmortem

#179

Post by tbodt »

abbeyj wrote: post #163
dnautics wrote: post #74

Earlier quoted context omitted.

> Debugging the problem was significantly hampered by failure of tools competing over use of the now-congested network. Man that's got to suck.

This reminded me of https://www.usenix.org/system/files/1311_05-08_mickens.pdf I HAVE NO TOOLS BECAUSE I’VE DESTROYED MY TOOLS WITH MY TOOLS.

You joke, but I've read a Google postmortem with that exact quote at the top. This incident would be more fitting though.

Re: Google Cloud Networking Incident Postmortem

#180

Post by ikiris »

tbodt wrote: post #179
abbeyj wrote: post #163

Earlier quoted context omitted.

This reminded me of https://www.usenix.org/system/files/1311_05-08_mickens.pdf I HAVE NO TOOLS BECAUSE I’VE DESTROYED MY TOOLS WITH MY TOOLS.

You joke, but I've read a Google postmortem with that exact quote at the top. This incident would be more fitting though.

It's in this one now.
Post reply on HN