481–490 of 539 posts
When Google said GCP is "down", did it affect entire availability zones within a region? For people who designed redundant infrastructure, did your backup AZs/regions keep your systems online?
Earlier quoted context omitted.
From the Cloudflare incident: > Cloudflare’s critical Workers KV service went offline due to an outage of a 3rd party service that is a key dependency. As a result, certain Cloudflare products that rely on KV service to store and disseminate information are unavailable [...] Surprising, but not entirely unplausible for a GCP outage to spread to CF.
> outage of a 3rd party service that is a key dependency. Good to know that Cloudflare has services seemingly based on GCP with no redundancy.
Earlier quoted context omitted.
Seems like a major wtf if Cloudflare is using GCP as a key dependency.
Some day Cloudflare will depend on GCP and GCP will depend on Cloudflare and AWS will rely on one of the two being online and Cloudflare will also depend on AWS and the internet will go down and no one will know how to restart it
Supposedly there are plans for how to conduct a "cold" start of the system, but as far as I know it's never actually been tried.
What's crazy is that RCS messaging is down as a result of this outage. It shows how poorly the technology or infrastructure was designed.
When Google said GCP is "down", did it affect entire availability zones within a region? For people who designed redundant infrastructure, did your backup AZs/regions keep your systems online?
To give an example, our web servers connect to our GCP CloudSQL database via a Cloud SQL Auth Proxy (there's also a connection pooler in between but that also stayed up). The connection to the proxy was always available, but the proxy wasn't able to renew auth tokens it uses to tunnel to the database, regardless of where the webserver or database happened to be located. To mitigate this in the future we're planning to stop using the auth proxy and connect directly via mutual TLS but now it means we have to manage TLS certificates.
Earlier quoted context omitted.
What's the alternative here? Do you want them to replicate their infrastructure across different cloud providers with automatic fail-over? That sounds -- heck -- I don't know if modern devops is really up to that. It would probably cause more problems than it would solve...
They're a company that has to run their own datacenters, you'd expect them to not fall over when a public cloud does.
Earlier quoted context omitted.
> outage of a 3rd party service that is a key dependency. Good to know that Cloudflare has services seemingly based on GCP with no redundancy.
What's the alternative here? Do you want them to replicate their infrastructure across different cloud providers with automatic fail-over? That sounds -- heck -- I don't know if modern devops is really up to that. It would probably cause more problems than it would solve...
Earlier quoted context omitted.
it's the best
Oh no, that's how you know it's nearing the point of being reaped and thrown in the graveyard!
Earlier quoted context omitted.
From the Cloudflare incident: > Cloudflare’s critical Workers KV service went offline due to an outage of a 3rd party service that is a key dependency. As a result, certain Cloudflare products that rely on KV service to store and disseminate information are unavailable [...] Surprising, but not entirely unplausible for a GCP outage to spread to CF.
> outage of a 3rd party service that is a key dependency. Good to know that Cloudflare has services seemingly based on GCP with no redundancy.
Earlier quoted context omitted.
SMS is pretty much decentralized, although there's a few companies with a lot of reach. I don't remember any Global SMS outages, but it wasn't uncommon for a whole carrier to have an SMS outage and especially for inter-carrier SMS to be broken from time to time (sometimes for days). I've certainly seen some stuff with SMS aggregators: almost all of them claim a majority of direct links, but when you have accounts wit…
> recently iOS started using it although I don't know what server they attach to. According to Wikipedia, only the carrier's RCS server is used [1] [1]: https://en.wikipedia.org/wiki/Rich_Communication_Services#So...