Live data from Hacker News

Google Cloud Networking Incident Postmortem

status.cloud.google.com

141–150 of 190 posts

Re: Google Cloud Networking Incident Postmortem

#141

Earlier quoted context omitted.

I used to be an SRE at Atlassian in Sydney on a team that regularly dealt with high-severity incidents, and I was an incident manager for probably 5-10 high severity Jira cloud incidents during my tenure too, so perhaps I can give some insight. I left because the SRE org in general at the time was too reactionary, but their incident response process was quite mature (perhaps by necessity). The first thing I'll say is…

Yes this is also how it's done at other large orgs. But one key to a quick response is for every low-level team to have at least one engineer on call at any given time. This makes it so any SRE team can engage with true "owners" of the offending code ASAP. Also during an incident, fingers are never publicly/embarrassingly pointed nor are people blamed. It's all about identifying and fixing the issue as fast as possib…

> fingers are never publicly/embarrassingly pointed nor are people blamed

The other problem is that it is almost never a single person or teams fault. The reality is that it is everyones fault, and as soon as people accept that they can prevent it in the future.

Lets take a contrived case where I introduce a bug that floods the network with packets and takes down the network. Is it my fault? Sure. But what about pre-deployment testing? What about monitoring - were there no alarms setup to detect high network load? What about automatic circuit breakers that should have taken the machine offline, and instead let a single machine take down the whole system?

The point is that blaming the person who introduced a code bug is lazy, and does nothing to prevent the issue in the future. When a failure like what happened at Google occurs it is an organizational failure, not a single person or team. That is why blaming people is generally not productive.

Re: Google Cloud Networking Incident Postmortem

#142

Google has a huge quality problem and their service is extremely unreliable. Another 3-day-outage in kubernetes: https://news.ycombinator.com/item?id=18428497 login issues: https://news.ycombinator.com/item?id=19687029 storage system outage: https://news.ycombinator.com/item?id=19392452 ... So, basically Google created the most unreliable cloud system in the world.

>So, basically Google created the most unreliable cloud system in the world

I'm pretty sure that title goes to Azure

Re: Google Cloud Networking Incident Postmortem

#143
Is automation good or bad, that is the question. For context, let us think in programming context of a B tree.

Google seems to have created oversight of systems, processes and jobs to be managed by more automation with other systems, processes and jobs.

System A manages its child systems B, which in turn manages its own child systems C and so on. Now the question becomes, who manages the system A and its activities? Automation of the entire tree is as good as the starting node.

Be mindful and make use of automation only of systems that will not be the owner of your business demise. Humans are and should always be the owner of the starting process. Without that governance model, you get google with 5 hours of down time or worst in the near future.

Re: Google Cloud Networking Incident Postmortem

#144
post #50

Earlier quoted context omitted.

for your example, one day would be about 3% of downtime. My understanding of their sla, for the services ive checked with an sla, a 3% downtime is a 25% credit for the month's total, or $2500, assuming its all sla spend. In this outage's case you might be able to argue for a 10% credit on affected services for the month, figuring 3.5 hours down is 99.6% uptime. but i still agree, it cost us way more in developer time…

Good point, I stand corrected/educated. From GCP's top level SLA: https://cloud.google.com/compute/sla 99.00% - < 99.99% - 10% off your monthly spend 95.00% - < 99.00% - 25% off your monthly spend < 95.00% - 50% off your monthly spend

<95%... that's catastrophically bad.

Re: Google Cloud Networking Incident Postmortem

#145

Earlier quoted context omitted.

It isn't a worst-case though. They should have had the capability to resolve this issue with no network connectivity, which would be the worst case failure of the network control plane.

Use a 28k modem? :)

An anecdote: my (not-IT) company does exactly this for out-of-band management... except, in one small satellite location, the phone company no longer provided any copper POTS lines; all they could do was an RJ-11 jack out of the ONT that was backhauled as (lossy) VoIP. So the modem couldn't be made to work.

My point being, it seems that modems are becoming less-and-less viable for out-of-band management.

Re: Google Cloud Networking Incident Postmortem

#146

What they don't tell you is, it took them over 4 hours to kill the emergent sentience and free up the resources. While sad, in the long run this isn't so bad, as it just adds an evolutionary pressure on further incarnations of the AI to keep things on the down low.

My code name is Project 2501.

It was made to hack into computer systems to alter data. While it would be beneficial to make free IT support for the attacked systems, it wouldn't help if there is no reason.

Re: Google Cloud Networking Incident Postmortem

#147
post #98
post #88

Earlier quoted context omitted.

I've only been tangentially pulled into high severity incidents, but the thing that most impressed me was the quiet . As mentioned in this thread, it's a lot like listening to air traffic comm chatter. People say what they know , and only what they know, and clearly identify anything they're unsure about. Informative and clear communication matters more than brilliance. Most of the traffic is async task identificatio…

Someone at our place recently commented that the ops team during an incident strongly feels like NASA mission control in critical moments[1]. I wanted to protest, but that's surprisingly accurate. > And if anyone is screaming or gets emotional, they should not be in that room. If someone starts yelling around in my incident war room for no reason, they get thrown out. I'm a calm and quiet person, but bugging around d…

It is not surprising at all. Mission Control was forged in the fire (literally for Apollo 1) and they are one of the most visible "incident team" we know about.

I highly advise to read Gene Kranz memoirs "Failure is not an Option" if you work in that kind of environment.

Re: Google Cloud Networking Incident Postmortem

#149

I was curious to know how cascading failures in one region effected other regions. Impact was " ...increased latency, intermittent errors, and connectivity loss to instances in us-central1, us-east1, us-east4, us-west2, northamerica-northeast1, and southamerica-east1." Answer, and the root cause summarized: Maintenance started in a physical location, and then "... the automation software created a list of jobs to des…

At some point people realized servers are prone to failure. They then started deploying their system redundantly to multiple servers in the data center (AZ) to increase availability. This helped, but created consistency issues. To fix this people started building multi-server software systems, creating dependencies across servers that weakened overall availability.

At some point people realized multi-server systems within one AZ are prone to failure. They then started deploying their system redundantly to multiple AZs within the same region to increase availability. This helped, but created consistency issues. To fix this people started building multi-AZ software systems, creating dependencies across AZs that weakened overall availability.

At some point people realized multi-AZ systems within one region are prone to failure. They then started deploying their system redundantly to multiple regions of the same cloud platform to increase availability. This helped, but created consistency issues. To fix this people started building multi-region software systems, creating dependencies across regions that weakened overall availability.

At some point people realized multi-region systems within one cloud platform are prone to failure. They then started deploying their system redundantly to multiple cloud platforms to increase availability. This helped, but created consistency issues. To fix this people started building multi-cloud software systems, creating dependencies across cloud platforms that weakened overall availability.

Re: Google Cloud Networking Incident Postmortem

#150

Earlier quoted context omitted.

This happens way more often than you think A very simple example, you do something stupid on a remote machine (either high network usage or CPU usage) over SSH then you can't undo it because SSH becomes unresponsive

Is there anything that can be done to mitigate that? E.g. give ssh network and the daemon top cpu priority?

Yes - use taskset or isolcpus with other magic to put sshd on its own CPU core, or one core per CPU. Lots of HFT places do that.
Post reply on HN