441–450 of 539 posts
Just our bi-yearly reminder of our over reliance on cloud providers for literally everything. Can't say there's an answer beyond trying to build more independent tech but we know how that goes.
Everything appears to be down as of 18:43 UTC... https://downdetector.com/
Any chance this is the root being that so many different services are effected? https://github.com/kubernetes/kops/issues/17433
Earlier quoted context omitted.
Down detector has a problem when whole clouds go down: unexpected dependencies. You see an app on a non-problematic cloud is having trouble, and report it to Down Detector but that cloud is actually fine- their actual stuff is running fine. What is really happening is that the app you are using has a dependency on a different SaaS provider who runs on the problematic cloud, and that is killing them. It's often things…
Surely if you build a status detector you realize that colo or dedicated are your only options, no? Obviously you cannot host such a service in the cloud.
My apps run on AWS, but we use third parties for logging, for auth support, billing, things like that. Some of those could well be on GCP though we didn't see any elevated error rates. Our system is resilient against those being down- after a couple of failed tries to connect it will dump what it was trying to send into a dump file for later re-sending. Most engineers will do that. But I've learned after many bad experiences that after a certain threshold of failures to connect to one of these outside system, my system should just skip calling out except for once every retryCycleTime, because all it will do is add two connectionTimeout's to every processing loop, building up messages in the processing queue, which eventually create backpressure up to the user. If you don't have that level of circuit breaker built, you can cause your own systems to give out higher error rates even if you are on an unaffected cloud.
So today a whole lot of systems that are not on GCP discovered the importance of the circuit breaker design pattern.
https://status.cloud.google.com/incidents/ow5i3PPK96RduMcb1S... > Multiple GCP products are experiencing impact due to Identity and Access Management Service Issue IAM issue huh. The post-mortem should be interesting at least.
Ha. With all this soviet style euphemism I rather read the onion instead.
Even a few years ago senior management knew to stay the fuck out except for asking for more info.