421–430 of 539 posts
Earlier quoted context omitted.
Because they have unrealistic targets so they make up fake uptime numbers. 99.999% would mean not even having an hour of downtime in 10 years. I remember reddit being down for like a whole day or so and they claimed 99.5% in that month.
Ma Bell hit that decently often.
It's a lot easier to keep packets flowing than to keep non-self-contained servers serving.
Earlier quoted context omitted.
Ma Bell hit that decently often.
Is that even knowable? Like, I know they called it “The Astonishing, Unfailing, Bell System” but if they had an outage somewhere did they actually have an infrastructure of “canary phones” and such to tell in real time? (As in, they’d know even if service was restored in an hour) Not trying to snark, I legit got nerdsniped by this comment.
Engineering and Operations in the Bell System is pretty great for this.
Earlier quoted context omitted.
Absolutely possible. Though there is something curious: https://www.cloudflarestatus.com/ At Cloudflare it started with: "Investigating - Cloudflare engineering is investigating an issue causing Access authentication to fail.". So this would somehow validate the theory of auth/quotas started failing right after Google, but what happened after ?! Pure snowballing ? That sounds a bit crazy.
Doesn't cloudflare have its own infrastructure, it's wild to me that both these things are down presumably together with this size of a blast radius.
As such, I wouldn't be overly surprised if all of CF's non-edge compute (including, for example, their control plane) is just tossed onto a "competitor" cloud like GCP. To CF, that infra is neither a revenue center, nor a huge cost center worth OpEx-optimizing through vertical integration.
Can't reach my nest thermometer, but their status page says it's fine lol
Earlier quoted context omitted.
Yep. Self-reporting status pages are pretty near worthless. At my former large company (not FAANG), we weren't allowed to update the status page until we got VP approval, which also required approval from both PR and Legal. It would take a lot more time and effort to get those approvals than to just fix the problem and move on.
SLA contracts, clawbacks, and performance obligations make these pages a bit of a minefield for CSPs. When I was at a top-tier CSP, we had the status page that was public, one that was for a trusted tier of customers, one built for a customer-by-customer basis, and one for internal engineering.
Cloudflare is down too. From https://www.cloudflarestatus.com : Update - We are seeing a number of services suffer intermittent failures. We are continuing to investigate this and we will update this list as we assess the impact on a per-service level. Impacted services: Access WARP Durable Objects (SQLite backed Durable Objects only) Workers KV Realtime Workers AI Stream Parts of the Cloudflare dashboard Jun 12, 202…
Everything appears to be down as of 18:43 UTC... https://downdetector.com/
Yeah. This service was presenting charts likely probed from inside GCP. I was on a call with a Google rep, someone pointed out that "AWS is also down" and I foolishly said something about "possible BGP attack" out of spite, before checking AWS availability myself. Shame on me.
Earlier quoted context omitted.
Because a lot of the time, not everyone is impacted, as the systems are designed to contain the "blast radius" of failures using techniques such as cellular architecture and [shuffle sharding]( https://aws.amazon.com/builders-library/workload-isolation-u... ). So sometimes a service is completely down for some customers and fully unaffected for other customers.
"there is a 5% chance your instance is down" is still a partial outage. A green check should only mean everything (about that service) is working for everyone (in that region) as intended. Downdetector reports started spiking over an hour ago but there still isn't a single status that isn't a green checkmark on the status page.
Earlier quoted context omitted.
You'd think so wouldn't you? DownDetector also reports azure and oracle cloud, I can't see then also being dependant on GCP... I guess down detector isn't a full source of truth though. https://ocistatus.oraclecloud.com/#/ https://azure.status.microsoft/en-gb/status Both green
Down detector has a problem when whole clouds go down: unexpected dependencies. You see an app on a non-problematic cloud is having trouble, and report it to Down Detector but that cloud is actually fine- their actual stuff is running fine. What is really happening is that the app you are using has a dependency on a different SaaS provider who runs on the problematic cloud, and that is killing them. It's often things…
Earlier quoted context omitted.
Because a lot of the time, not everyone is impacted, as the systems are designed to contain the "blast radius" of failures using techniques such as cellular architecture and [shuffle sharding]( https://aws.amazon.com/builders-library/workload-isolation-u... ). So sometimes a service is completely down for some customers and fully unaffected for other customers.
> Because a lot of the time, not everyone is impacted then such pages should report a partial failure. Indeed the GCP outage page lists an orange "One or more regions affected" marker, but all services show the green "Available" marker, which apparently is not true.