This is why any criticism of AWS reliability is meaningless to me. All the cloud providers go down - all of them. Either you are multi-cloud, or you run your own hardware, but these events are inevitable.
Ongoing Incident in Google Cloud
31–40 of 115 posts
Re: Ongoing Incident in Google Cloud
#32This is why any criticism of AWS reliability is meaningless to me. All the cloud providers go down - all of them. Either you are multi-cloud, or you run your own hardware, but these events are inevitable.
Inevitable != immune to criticism
Re: Ongoing Incident in Google Cloud
#33https://packages.cloud.google.com/apt/doc/apt-key.gpg Even the public apt key for signing Google's cloud packages is unavailable (returns 500 for me). This is insane
[deleted]
Re: Ongoing Incident in Google Cloud
#3405:41 - 06:26 PT, 45 min total. Not great, not terrible.
Yep. Of course there's no detail yet so we don't know what exactly was affected. All we can see is "Multiple services are being impacted globally" and a list of services (Build, Firestore, Container Registry, BigQuery, Bigtable, Networking, Pub/Sub, Storage, Compute Engine, Identity and Access Management) but there's no indication of what specifically was impacted. Could you still see status for your VMs, but not launch new ones? Was it mostly affecting only a couple regions? No idea. All we know is they're now below four nines in February for a handful of critical services.
Let's take a gander at incident history: https://status.cloud.google.com/summary
Cloud Build looks bad... three multi-hour incidents this year, four in fall/winter last year.
Cloud Developer Tools have had four multi-hour incidents this year, many last fall/winter.
Cloud Firestore looks abysmal... Six multi-hour incidents this year, one of them 23 hours.
Cloud App Engine had three multi-hour incidents this year, many in fall/winter last year.
BigQuery had three multi-hour incidents this year, many in fall/winter last year.
Cloud Console had five multi-hour incidents this year, many in fall/winter last year. (And from my personal experience, their console blows pretty much all the time)
Cloud Networking has had nine incidents this year, one of them was eight days long. What the fuck.
Compute Engine has had five multi-hour incidents this year, many last fall/winter.
GKE had 3 incidents this year, multiple the past winter.
Can somebody do a comparison to AWS? This seems shitty but maybe it's par for the course?
Re: Ongoing Incident in Google Cloud
#35This is why any criticism of AWS reliability is meaningless to me. All the cloud providers go down - all of them. Either you are multi-cloud, or you run your own hardware, but these events are inevitable.
> you run your own hardware
in multiple datacenters?
Re: Ongoing Incident in Google Cloud
#36This is why any criticism of AWS reliability is meaningless to me. All the cloud providers go down - all of them. Either you are multi-cloud, or you run your own hardware, but these events are inevitable.
This is why any criticism of AWS > reliability is meaningless to me.
Er, we absolutely can and should compare rates of problems and overall reliability.
Re: Ongoing Incident in Google Cloud
#37This demonstrates yet again why global configurations, global services, and global anycast VIP routing should be considered an anti pattern. gcp should be designed in a way where the term “global outage” isn’t a word in their vocabulary.
You can't really have 30+ fully independent regions running their own stack with different versions of apps and separate secrets, IP/routing and certificates in each. At some point you have to unify or it becomes either unmanageable or inconsistent.
[dead]
Re: Ongoing Incident in Google Cloud
#38This demonstrates yet again why global configurations, global services, and global anycast VIP routing should be considered an anti pattern. gcp should be designed in a way where the term “global outage” isn’t a word in their vocabulary.
> gcp should be designed in a way where the term “global outage” isn’t a word in their vocabulary. As I understand it, GCP is already designed to make global outages impossible. Obviously this outage shows that they messed up somehow and some global point of failure still remains. Looking forward to the post-mortem.
They had many many global outages through the years so that’s evidently not true. GCLB, iam, gcs and probably more Im missing just of the top of my head. Then there’s constant stream of regional networking borks where your latency is suddenly 5x which are not “global” but affect multiple regions
Re: Ongoing Incident in Google Cloud
#39This demonstrates yet again why global configurations, global services, and global anycast VIP routing should be considered an anti pattern. gcp should be designed in a way where the term “global outage” isn’t a word in their vocabulary.
there are three things that scare google engineers enough to keep them up at night: a global network outage, a global power outage, and a global chubby outage. Actually, they only really worry about that last one.
Re: Ongoing Incident in Google Cloud
#40Ouch some pain at google today then. I hate to wake up on a Monday morning to this. <3 To the engineers trying to fix it at the moment.
Google has follows-the-sun on-call rotations for large rotations, so this hit the UK team just after lunch.