Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Ask HN: Why are most status pages delayed?

news.ycombinator.com

Post reply on HN
Print view View original

31–40 of 62 posts

Re: Ask HN: Why are most status pages delayed?

#31

Post by FinnKuhn »

anomaloustho wrote: post #26

It’s already been said, but most companies already have those instant “alarms” that go off within minutes. 80% of the time, those alarms are red herrings that get triaged. At a lot of companies, they go off constantly. As a company, you don’t want to declare an outage readily and you definitely don’t want it to be declared frequently. Declaring an outage frequently means: • Telling your exec team that your department…

>So every social and incentive structure along the way basically signals that you don’t want to declare an outage when it isn’t real.

I would argue that every social and incentive structure along the way basically signals that you don't want to declare an outage, even when it is real. You should still do it though or it becomes meaningless.

Great example for Goodhart's law.

Re: Ask HN: Why are most status pages delayed?

#32

Post by sjsdaiuasgdia »

Status pages usually start as a human-updated thing because it's easy to implement.

Some time later, you might add an automated check thing that makes some synthetic requests to the service and validates what's returned. And maybe you wire that directly to the status page so issues can be shown as soon as possible.

Then, false alarms happen. Maybe someone forgot to rotate the credentials for the test account and it got locked out. Maybe the testing system has a bug. Maybe a change is pushed to the service that changes the output such that the test thinks the result is invalid. Maybe a localized infrastructure problem is preventing the testing system from reaching the service. There's a lot of ways for false alarms to appear, some intermittent and some persistent.

So then you spread out. You add more testing systems in diverse locations. You require some N of M tests to fail, and if that threshold is reached the status page gets updated automatically. That protects you from a few categories of false alarms, but not all of them.

You could go further to continue whacking away at the false alarm sources, but as you go you run into the same problem of service reliability, where each additional "9" costs much more than the one that came before. You reach a point where you realize the cost of making your automatic status page updates fully automatically accurate becomes prohibitive.

So you go back to having a human assess the alarm and authorize a status page update if it is legitimate.

Re: Ask HN: Why are most status pages delayed?

#33

Post by gwbas1c »

2gremlin181 wrote: post #9
knorker wrote: post #5

To add to the reasons others gave: It needs to be correct. Engineers are working the problem. They have a pretty good understanding of the impact of the outage. Then an external comms person asks for an engineer to proof read the external outage comms. Which triggers rounds of "no, this part is not technically correct" and "I know the internal system scope impact, but not how that maps to external product names you w…

I totally get that, but how hard would it be to actually make calls to your own API from the status page? If it fails, display a vague message saying there might be issues and that you are looking into it. Clearly these metrics and alerts exist internally too. I'm not asking for an instant RCA or confirmation of the scope of the outage. Just stop gaslighting me.

Automation is always significantly easier said than done. Furthermore, as has been emphasized elsewhere in this thread, there is a critical need for a human in the loop.

Re: Ask HN: Why are most status pages delayed?

#34

Post by mirekrusin »

Because they're incentivized to delay it, ideally until resolved, this way their SLA uptime is 100%. Less of reported downtime is better for them so they push it as much as possible. If they were to report all failures their pretty green history would be filled with red. What, are you going to do, sue them? They can do it so they do.

Re: Ask HN: Why are most status pages delayed?

#35

Post by gwbas1c »

anomaloustho wrote: post #26

It’s already been said, but most companies already have those instant “alarms” that go off within minutes. 80% of the time, those alarms are red herrings that get triaged. At a lot of companies, they go off constantly. As a company, you don’t want to declare an outage readily and you definitely don’t want it to be declared frequently. Declaring an outage frequently means: • Telling your exec team that your department…

Just wanted to chime in that, at my company, we have some policies that impact when we actually update our status page to show that we have an outage. Without going into details, the policies deliberately slow down our reporting of downtime: We (engineering) need to have a clear understanding of what the problem is before we say there is a problem publicly.

I've personally challenged some details in these policies, which I won't discuss publicly. What I generally agree with is that it's important to have a human in the loop, and to be very thoughtful about when to update a status page and what is put there.

Re: Ask HN: Why are most status pages delayed?

#37

Post by [deleted] »

swiftcoder wrote: post #15

Because for most major sites, updating the status page requires (a significant number of) humans in the loop. Back when I worked at a major cloud provider (which admittedly was >5 years ago), our alarms would go off after ~3-15 minutes of degraded functionality (depending on the sensitivity settings of that specific alarm). At that point the on call gets paged in to investigate and validates that the issue is real (a…

[deleted]

Re: Ask HN: Why are most status pages delayed?

#38

Post by swiftcoder »

2gremlin181 wrote: post #17
swiftcoder wrote: post #15

Because for most major sites, updating the status page requires (a significant number of) humans in the loop. Back when I worked at a major cloud provider (which admittedly was >5 years ago), our alarms would go off after ~3-15 minutes of degraded functionality (depending on the sensitivity settings of that specific alarm). At that point the on call gets paged in to investigate and validates that the issue is real (a…

Copying my response over from another comment: I totally get that, but how hard would it be to actually make calls to your own API from the status page? If it fails, display a vague message saying there might be issues and that you are looking into it. Clearly these metrics and alerts exist internally too. I'm not asking for an instant RCA or confirmation of the scope of the outage. Just stop gaslighting me.

Not hard at all (our internal dashboards did just that). But to have that data posted publicly was not in the best interests of the business.

And honestly, having been on a few customer escalations where they threatened legal action over outages, one kind of starts to see things the business way...

Re: Ask HN: Why are most status pages delayed?

#39

Post by mirekrusin »

swiftcoder wrote: post #15

Because for most major sites, updating the status page requires (a significant number of) humans in the loop. Back when I worked at a major cloud provider (which admittedly was >5 years ago), our alarms would go off after ~3-15 minutes of degraded functionality (depending on the sensitivity settings of that specific alarm). At that point the on call gets paged in to investigate and validates that the issue is real (a…

So there is access to "degraded functionality" from start (the "3-15" of "degraded functionality" one) - people are asking why not share THAT then?

Nobody cares about internal escalations, if manager is taking shit or not - that's not service status, that's internal dealing with the shit process - it can surface as extra timestamped comments next to service STATUS.

Re: Ask HN: Why are most status pages delayed?

#40

Post by swiftcoder »

mirekrusin wrote: post #39
swiftcoder wrote: post #15

Because for most major sites, updating the status page requires (a significant number of) humans in the loop. Back when I worked at a major cloud provider (which admittedly was >5 years ago), our alarms would go off after ~3-15 minutes of degraded functionality (depending on the sensitivity settings of that specific alarm). At that point the on call gets paged in to investigate and validates that the issue is real (a…

So there is access to "degraded functionality" from start (the "3-15" of "degraded functionality" one) - people are asking why not share THAT then? Nobody cares about internal escalations, if manager is taking shit or not - that's not service status, that's internal dealing with the shit process - it can surface as extra timestamped comments next to service STATUS.

> why not share THAT then?

When you've guaranteed 4 or 5 nines worth of uptime to the customer, every acknowledged outage results in refunds (and potentially being sued over breach of contract)

Post reply on HN