24h time format used in Postmortem. Interesting.
It's the superior format. Just like yyyy-mm-dd [hh:mm:ss.sss] is, because lexicographic string order matches the time order.
Google Cloud Networking Incident Postmortem
131–140 of 190 posts
Re: Google Cloud Networking Incident Postmortem
#132Google has a huge quality problem and their service is extremely unreliable. Another 3-day-outage in kubernetes: https://news.ycombinator.com/item?id=18428497 login issues: https://news.ycombinator.com/item?id=19687029 storage system outage: https://news.ycombinator.com/item?id=19392452 ... So, basically Google created the most unreliable cloud system in the world.
They only have big outages. The VMs are incredibly reliable other than the big incidents. And, as I am often reminded by my product owners, people don't mind big outages as much as much as small random failures. If the whole thing is down, ok, fine, I'll go home. If it fails 0.1% all the time, my life is suffering. And in GCP, you start a VM and it just stays up. We've killed them from inside with memory leaks and filling the disk etc., but I haven't seen GCP kill them (50K VMs for couple years).
Re: Google Cloud Networking Incident Postmortem
#133Earlier quoted context omitted.
You have the gist I would say. It's important to understand that Google separates the control plane and data plane, so if you think of the internet, routing tables and bgp are the control part and the hardware, switching, and links are data plane. Often times those two are combined in one device. At Google, they are not. So the part that sets up the routing tables talking to some global network service went down. The…
It shouldn't. Amazon believes in strict regional isolation, which means that outages only impact 1 region and not multiple. They also stagger their releases across regions to minimize the impact of any breaking changes (however unexptected...)
While I agree it sounds like their networking modules cross-talk too much - you still need to store the networking config in some single global service (like a code version control system). And you do need to share across regions some information on cross-region link utilization.
Re: Google Cloud Networking Incident Postmortem
#134Earlier quoted context omitted.
> Debugging the problem was significantly hampered by failure of tools competing over use of the now-congested network. Man that's got to suck.
This happens way more often than you think A very simple example, you do something stupid on a remote machine (either high network usage or CPU usage) over SSH then you can't undo it because SSH becomes unresponsive
Yeah got locked out of a dedi this way (bad ipfw ruleset which killed my ssh connection) and the virtual console wasn’t working. Fun times. At least it was a personal machine.
Re: Google Cloud Networking Incident Postmortem
#135I was curious to know how cascading failures in one region effected other regions. Impact was " ...increased latency, intermittent errors, and connectivity loss to instances in us-central1, us-east1, us-east4, us-west2, northamerica-northeast1, and southamerica-east1." Answer, and the root cause summarized: Maintenance started in a physical location, and then "... the automation software created a list of jobs to des…
Said satirical note on Byzantine fault tolerance is on this link [0]. As usual for Mickens, gives that "funny, but true" sense.
[0] https://scholar.harvard.edu/files/mickens/files/thesaddestmo...
Re: Google Cloud Networking Incident Postmortem
#136(meme) I figured out why the google outage took a while to recover: https://i.imgur.com/hzcLx5X.png
Re: Google Cloud Networking Incident Postmortem
#137Earlier quoted context omitted.
> Debugging the problem was significantly hampered by failure of tools competing over use of the now-congested network. Man that's got to suck.
It isn't a worst-case though. They should have had the capability to resolve this issue with no network connectivity, which would be the worst case failure of the network control plane.
Use a 28k modem? :)
Re: Google Cloud Networking Incident Postmortem
#138Given the fact that the status page was reporting for more than 30 minutes an erroneous infrastructure state and this is google, is it okay for Amazon to put the SRE books into the "Science Fiction" category or should we keep them under tech?
I still feel for the on-call engineers.
Re: Google Cloud Networking Incident Postmortem
#139The HA/Scheduling system is too complex.
Simplify it Google!
Re: Google Cloud Networking Incident Postmortem
#140Earlier quoted context omitted.
There is this from May from Network World: https://www.networkworld.com/article/3394341/when-it-comes-t... GCP was basically even with AWS, and Microsoft was ~6x their downtime according to that article.
From the article: > AWS has the most granular reporting, as it shows every service in every region. If an incident occurs that impacts three services, all three of those services would light up red. If those were unavailable for one hour, AWS would record three hours of downtime. Was this reflected in their bar graph or not? Also, GCP has had a number of global events, e.g. the inability to modify any load balancer f…
While I would like to say AWS hasn't had that issue, in 2017 it did (just not because of load balancers being unavailable, but as a consequence of the S3 outage [1].
When the primary S3 nodes went down, it caused connectivity issues to S3 buckets globally, and services like RDS, SES, SQS, Load Balancers, etc etc, all relied on getting config information from the "hidden" S3 buckets, thus people couldn't edit load balancers.
(Outage also meant they couldn't update their own status page! [2])
[1]: https://aws.amazon.com/message/41926/ [2]: https://www.theregister.co.uk/2017/03/01/aws_s3_outage/