if all services at down at once, no one is thinking or mentioning a potential attack on US cloud providers ? (China or Russia) Maybe ?
GCP Outage
271–280 of 539 posts
Re: GCP Outage
#272It's completely nuts that Firebase has this: https://status.firebase.google.com/incidents/ZcF1YDUvpdixZ2e... "Firebase Data Connect unavailable due to a known Google Cloud global outage" While the Google Cloud status page https://status.cloud.google.com/ says "No major incidents" and everything is green. So Google Cloud know there is an outage but just deem it not major enough to show it. Edit to add: within 10 minut…
IIRC status pages drive customer compensation for downtime. Updating it is basically signing the check for their biggest customers, in most similar companies you need a very senior executive to approve the update On the other side of this, Firebase probably doesn't have money at stake making the update
Nah, its just some client side caching / JS stuff. Clicking the big refresh button fixed it for me, 15 minutes before OP noted it.
(n.b. as much as Google in aggregate is evil, they're smart evil. You can't avoid execs approving every outage because checks without some paper trail, and execs don't want to approve every outage, you'd have to rely on too many engineers and sales people, even as ex-employees, to keep it a secret. disclaimer: xoogler)
(EDIT: for posterity, we're discussing a "overall status" thing with a huge refresh button, right above a huge table chockful of orange triangles that indicate "One or more regions affected" - even when the "overall status" was green, the table was still full of orange and visible immediately underneath. My point being, you gotta suppose a wholeeee bunch of stuff to get to the point there was ever info suppressed, much less suppressed intentionally to avoid cutting checks)
Re: GCP Outage
#273reCAPTCHA affected? I couldn't log into my local utilities website due to a reCAPTCHA error. Downdetector agrees, but I interpret that site as dubious.
Re: GCP Outage
#274It's completely nuts that Firebase has this: https://status.firebase.google.com/incidents/ZcF1YDUvpdixZ2e... "Firebase Data Connect unavailable due to a known Google Cloud global outage" While the Google Cloud status page https://status.cloud.google.com/ says "No major incidents" and everything is green. So Google Cloud know there is an outage but just deem it not major enough to show it. Edit to add: within 10 minut…
Services are recovering in some locations it seems - Discord is healing
Re: GCP Outage
#275It's completely nuts that Firebase has this: https://status.firebase.google.com/incidents/ZcF1YDUvpdixZ2e... "Firebase Data Connect unavailable due to a known Google Cloud global outage" While the Google Cloud status page https://status.cloud.google.com/ says "No major incidents" and everything is green. So Google Cloud know there is an outage but just deem it not major enough to show it. Edit to add: within 10 minut…
AWS has this all the time. If you need to know if a service is down in a region, check for other engineers talking about it on X.
Re: GCP Outage
#276Earlier quoted context omitted.
This feels very much like when facebook, locked themselves out of their datacenters. ;) * https://www.datacenterdynamics.com/en/news/facebook-blames-m...
Except that AWS, CloudFlare and a bunch others are also down :-O
AWS looks ok to me?
https://health.aws.amazon.com/health/status
Perhaps CF is dependant on some GCP services?
Re: GCP Outage
#277Interesting that all Digital Ocean services are fine...
Re: GCP Outage
#278Earlier quoted context omitted.
IIRC status pages drive customer compensation for downtime. Updating it is basically signing the check for their biggest customers, in most similar companies you need a very senior executive to approve the update On the other side of this, Firebase probably doesn't have money at stake making the update
It is not the status page that drives customer compensation. It is downtime.
The status page is essentially an admission of guilt. It can require approval from the legal department and a high level official from the company to approve updating it and the verbiage used on the status page.
Re: GCP Outage
#279Earlier quoted context omitted.
That's the case with every status page. These pages are managed by business people not engineers, because their primary purpose is to show customers that the company is meeting contractually defied SLAs.
Surelly no SLA will be based on the display of the status page...
should* be
Re: GCP Outage
#280One of these days in which the young engineers learn the concept of 'counterparty risk'.