51–60 of 62 posts
We are now regularly detecting outages long before providers acknowledge them which is hugely beneficial to IT teams.
For this Reddit outage, we alerted 13 minutes before the official status page.
Last weeks Azure outage, it was 42 minutes prior (!?!).
To add to the reasons others gave: It needs to be correct. Engineers are working the problem. They have a pretty good understanding of the impact of the outage. Then an external comms person asks for an engineer to proof read the external outage comms. Which triggers rounds of "no, this part is not technically correct" and "I know the internal system scope impact, but not how that maps to external product names you w…
I totally get that, but how hard would it be to actually make calls to your own API from the status page? If it fails, display a vague message saying there might be issues and that you are looking into it. Clearly these metrics and alerts exist internally too. I'm not asking for an instant RCA or confirmation of the scope of the outage. Just stop gaslighting me.
Ah, so you're saying the status page should be hooked up to internal monitoring probers?
So how sure are you that it's the service that's broken, and not the probers? How sure are you that the granularity of the probers reflect the actual scope of the outage?
Also this opens up questioning of "well why don't you have probing on the EXACT workflow that happened to break this time?!". Because honestly, that's not helpful.
Say you have a complete end to end workflow for your web store. Should you publish "100% outage, the webstore is down!!" on your status page, automatically, because the very diligent prober failed to get into the shoe section of your store? That's probably not helpful to anybody.
> Clearly these metrics and alerts exist internally too.
Well, no. Probers can never cover every dimension across which a service can have an outage. You may think that the service is simple and has an obvious status, but you're using like 0.1% of the user surface, and have never even heard of the weird things that 99% of actual traffic does.
How do you even model your minority use case? Is it an outage? Or is your workflow maybe a tiny weird one, even though you think it's the straightforward one?
Especially since the nature of outages in complex systems tend to be complex to describe accurately. And a status page needs to boil it down to simple.
In many cases even engineers inspecting the system can not always be certain if real users are experiencing an outage, or if they're chasing an internal user, or if nothing is user visible because internal retries are taking care of everything, or what.
Complex systems are often complex because the world is complex. And if the problem is simple and unevolving then there would be no reason to have outages in the first place.
And often engineers helping phrase an outage statement need to compromise verbosity for clarity.
Another thing is what do you do if you start serving 500s to 90% of traffic? An outage, right? Surely auto-publish to a status page? Oh, but it turns out this was a DoS attack, and no non-DoS traffic was affected. Can your monitoring detect the difference? Unlikely.
I don't think it's a factor in how Statuspage works. Cloudflare, for example, uses them, and usually it's pretty fast to update their status page and release outage information.
For companies that need to monitor critical dependencies, my company ( https://isdown.app ) helps by aggregating status page information with crowdsourced reports. This way, companies can be alerted way sooner than when the status page is updated.
It’s not a technical issue, it’s a business one. Those status pages are often linked to contractual SLAs and updating the page tangibly means money lost. So there’s an incentive to only up it when the issue is severe and not quickly remediated. It’s not an engineers tool, it’s a liability tool.
Imagine what you could get away with if you owned the ledger of truth. Any time you made an offence, you could just update that ledger to say the people complaining are wrong, and that's the end of it. I feel that the tech industry does not have sole ownership of this powerful tool
Earlier quoted context omitted.
So there is access to "degraded functionality" from start (the "3-15" of "degraded functionality" one) - people are asking why not share THAT then? Nobody cares about internal escalations, if manager is taking shit or not - that's not service status, that's internal dealing with the shit process - it can surface as extra timestamped comments next to service STATUS.
> why not share THAT then? When you've guaranteed 4 or 5 nines worth of uptime to the customer, every acknowledged outage results in refunds (and potentially being sued over breach of contract)
By not showing metrics they actually had to go and complain manually to get refunds / adjustments.
Also, if a server is down at 2am when the clients aren't using it, is it really down?
Two main reasons: 1) The status page update is not automated. It takes time for the incident management process to go through all the steps and updating the status page is generally not the first one. It's often an overlooked one.
2) The company would rather not communicate that they are/were down, or they are not yet sure of the impact, and therefore don't yet have the correct information to share.
Of course, all modern incident platforms will offer tools to update your page super easily/in real-time. But tooling is generally not the main issue; it's process/information.