Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Ask HN: Why are most status pages delayed?

news.ycombinator.com

Post reply on HN
Print view View original

21–30 of 62 posts

Re: Ask HN: Why are most status pages delayed?

#21

Post by rozenmd »

2gremlin181 wrote: post #17
swiftcoder wrote: post #15

Because for most major sites, updating the status page requires (a significant number of) humans in the loop. Back when I worked at a major cloud provider (which admittedly was >5 years ago), our alarms would go off after ~3-15 minutes of degraded functionality (depending on the sensitivity settings of that specific alarm). At that point the on call gets paged in to investigate and validates that the issue is real (a…

Copying my response over from another comment: I totally get that, but how hard would it be to actually make calls to your own API from the status page? If it fails, display a vague message saying there might be issues and that you are looking into it. Clearly these metrics and alerts exist internally too. I'm not asking for an instant RCA or confirmation of the scope of the outage. Just stop gaslighting me.

There are increasingly more status pages that automatically update based on uptime data (I built a service providing that - OnlineOrNot)

But early-stage startups typically have engineering own the status page, but as they grow, ownership usually transfers to customer support. These teams optimize for controlling the message rather than technical detail, which explains the shift toward vaguer/slower incident descriptions.

Re: Ask HN: Why are most status pages delayed?

#22

Post by fouc »

hypeatei wrote: post #18

Why even check the status page at all if you're experiencing errors and others are reporting the same? I don't see the point in getting worked up over how long it takes a company to update their status page. There is a ton of moving pieces in software these days and networks in general. There is no straightforward way to declare an outage via health checks, especially if declaring said outage can cost you $$ due to v…

"A status page is used to communicate real-time information about a company's system health, performance, and any ongoing incidents to users. It helps reduce support tickets, improve transparency, and build trust by keeping users informed during outages or maintenance."

By definition, status pages should reflect reality, and be real-time, it makes sense to get worked up over because we rely on the company's status page to be the ultimate arbiter of the reality of their servers. It's not always easily obvious if it's on our end or theirs. Even if some other third party checker is showing problems, it doesn't mean they're 100% accurate either.

Re: Ask HN: Why are most status pages delayed?

#23

Post by fouc »

I agree, literally the definition of a status page is:

"A status page is used to communicate real-time information about a company's system health, performance, and any ongoing incidents to users. It helps reduce support tickets, improve transparency, and build trust by keeping users informed during outages or maintenance"

real-time. for multiple good reasons. reduces confusion for everyone.

Re: Ask HN: Why are most status pages delayed?

#24

Post by pixl97 »

hypeatei wrote: post #18

Why even check the status page at all if you're experiencing errors and others are reporting the same? I don't see the point in getting worked up over how long it takes a company to update their status page. There is a ton of moving pieces in software these days and networks in general. There is no straightforward way to declare an outage via health checks, especially if declaring said outage can cost you $$ due to v…

Because it takes work to figure out if others are having issues too or it's you with the problem. You're checking somethings status somewhere.

Re: Ask HN: Why are most status pages delayed?

#25

Post by jpalawaga »

It’s not a technical issue, it’s a business one.

Those status pages are often linked to contractual SLAs and updating the page tangibly means money lost.

So there’s an incentive to only up it when the issue is severe and not quickly remediated.

It’s not an engineers tool, it’s a liability tool.

Re: Ask HN: Why are most status pages delayed?

#26

Post by anomaloustho »

It’s already been said, but most companies already have those instant “alarms” that go off within minutes. 80% of the time, those alarms are red herrings that get triaged. At a lot of companies, they go off constantly.

As a company, you don’t want to declare an outage readily and you definitely don’t want it to be declared frequently. Declaring an outage frequently means:

• Telling your exec team that your department is not running well • Negative signal to your investors • Bad reputation with your customers • Admitting culpability to your customers and partners (inviting lawsuits and refunds) • Telling your engineering leadership team that your specific team isn’t running well • Messing up your quarterly goals, bonuses etcetera for outages that aren’t real

So every social and incentive structure along the way basically signals that you don’t want to declare an outage when it isn’t real. You want to make sure you get it right. Therefore, you don’t just want to flip a status page because a few API calls had a timeout.

Re: Ask HN: Why are most status pages delayed?

#28

Post by arccy »

Another way to look at it is: you already know the service is down because you can't use it. The status page being manually updated means someone is aware and actually working on fixing it, rather than it being automated and the other side just ignoring it...

Re: Ask HN: Why are most status pages delayed?

#29

Post by chrismorgan »

A few months ago, Cloudflare accidentally turned off 1.1.1.1 (I’m simplifying slightly, most notably DNS-over-HTTPS continued to work). Over the course of five or six minutes, traffic dropped to 10% of normal, and stayed there. Somehow, it took another six minutes before an alert fired, at which point they noticed.

https://news.ycombinator.com/item?id=44578490

You’d think that for such a company they’d notice if global traffic for one of their important services for a given minute had dropped below 50% compared with the last hour, but apparently not.

And that’s Cloudflare, who I would expect better of than most.

Re: Ask HN: Why are most status pages delayed?

#30

Post by FinnKuhn »

dvt wrote: post #20
2gremlin181 wrote: post #17

Earlier quoted context omitted.

Copying my response over from another comment: I totally get that, but how hard would it be to actually make calls to your own API from the status page? If it fails, display a vague message saying there might be issues and that you are looking into it. Clearly these metrics and alerts exist internally too. I'm not asking for an instant RCA or confirmation of the scope of the outage. Just stop gaslighting me.

> Just stop gaslighting me. I heard this years ago from someone, but there's material impact to a company's bottom line if those pages get updated, so that's why someone fairly senior has to usually "approve" it. Obviously it's technically trivial, but if they acknowledge downtime (for example, like in the AWS case), investors will have questions, it might make quarterly reports, and it might impact stock price. So i…

I feel like there should at least be some sort of disclaimer then that tells me the status page can take up to xx minutes to show an outage and not make it seem as if it is updated instantaniously. That way I could way those xx minutes before I file a ticket with support and not have the case thinking it is an isolated problem for me instead of a major outage.
Post reply on HN