Can't wait for the postmortem!
My money is on config push.
I would question why they are doing a rollout on a weekend?
Google Cloud Is Down
201–210 of 630 posts
Re: Google Cloud Is Down
#202Does anyone know if this is a regional or global outage? I can see my GKE clusters in one region but not in another, so I am guessing it's the former. Looks like we'll need a cluster in each region going forward...
I can not access vimeo from EU, not sure if related to Google though.
Re: Google Cloud Is Down
#203Earlier quoted context omitted.
You have my honest sympathy because of the difficulties you now suffer through, but it bears emphasizing: this is what you get when you replace what should be a physical product under your control with Internet-connected service running on third-party servers. IoT as seen on the consumer market is a Bad Idea.
It's a trade-off of risks. Leaving a key under the may could lead to a security breach.
At least you can isolate your security risk to something you have more control over than a random network outage.
Re: Google Cloud Is Down
#204Earlier quoted context omitted.
You have my honest sympathy because of the difficulties you now suffer through, but it bears emphasizing: this is what you get when you replace what should be a physical product under your control with Internet-connected service running on third-party servers. IoT as seen on the consumer market is a Bad Idea.
It's a trade-off of risks. Leaving a key under the may could lead to a security breach.
So can a compromise of a "smart" lock.
It's a fake trade-off, because you're choosing between lo-tech solution and bad engineering. IoT would work better if you made the "I" part stand for "Intranet", and kept the whole thing a product instead of a service. Alas, this wouldn't support user exploitation.
Re: Google Cloud Is Down
#205Earlier quoted context omitted.
This is weird, I had Zoom installed a long time ago but uninstalled according to their instructions [1]. I'm a macOS user. As soon as I clicked that link, the client downloaded a PKG file, installed itself and launched itself without asking me if I wanted to share my camera or audio. I uninstalled according to their instructions again, searched for all " zoom " files in my disk and rebooted. This leads me to believe…
Do you have auto-run enabled for downloaded files?
Nope. Not even for safe (image, music, etc) files.
And even if I had, PKG files don't install themselves on macOS, they open the installer interface, AFAIK.
EDIT: And, as I mentioned, the file was downloaded by some Zoom client, not by my browser.
Re: Google Cloud Is Down
#206Anyone experiencing issues with GCS? Seems highly intermittent and dependent on the location the request is coming from (maybe that's because it's a networking bug).
The status page says GCS is fine but that's highly unlikely.
Re: Google Cloud Is Down
#207Earlier quoted context omitted.
How come?
Solutions based on third-party butts have essentially two modes: the usual, where everything is smooth, and the bad one, where nothing works and you're shit out of luck - you can't get to your data anymore, because it's in my butt, accessible only through that butt, and arguably not even your data. With on-prem solutions, you can at least access the physical servers and get your data out to carry on with your day whi…
Any solution would be based on third parties, the robust solution is either to run your own country with fuel sources for electricity and army to defend the datacenters or rely on multiple independent infrastructures. I think the latter is less complex.
Re: Google Cloud Is Down
#208Earlier quoted context omitted.
AWS is often the same way. No one seems to be good at communicating outage details.
I really don’t get this. There’s a huge number of complaints about poor communication from companies like Google and AWS during every outage. Yet they remain seemingly indifferent to how much customer trust they are losing, and the competitive edge the first one to get this right could gain.
If 20% of Google Cloud's customers leave after this outage because of poor communication they'll prioritise accordingly and apply all that nice SRE theories to their infra. But this isn't happening, because , so... who cares?
Re: Google Cloud Is Down
#209Earlier quoted context omitted.
> including unfortunately the tooling we usually use to communicate across the company about outages. There's some irony in that.
AWS experienced a major outage a few years ago that couldn't be communicated to customers because it took out all the components central to update the status board. One of those obvious-in-hindsight situations. Not long after that incident, they migrated it to something that couldn't be affected by any outage. I imagine Google will probably do the same thing after this :)
Was just reading it, they made their status page multi-region.
Re: Google Cloud Is Down
#210Earlier quoted context omitted.
How come?
Solutions based on third-party butts have essentially two modes: the usual, where everything is smooth, and the bad one, where nothing works and you're shit out of luck - you can't get to your data anymore, because it's in my butt, accessible only through that butt, and arguably not even your data. With on-prem solutions, you can at least access the physical servers and get your data out to carry on with your day whi…
Assuming you have data that is tiny enough to fit anywhere other than the cluster you were using.
Assuming you can afford to have a second instance with enough compute just sitting around.
Assuming it's not the HDDs, RAID controller, SAN, etc which is causing the outage.
Assuming it's not a fire/flood/earthquake in your datacenter causing the outage.
...etc.