Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Ask HN: Why are most status pages delayed?

news.ycombinator.com

Post reply on HN
Print view View original

41–50 of 62 posts

Re: Ask HN: Why are most status pages delayed?

#41

Post by onionisafruit »

chrismorgan wrote: post #29

A few months ago, Cloudflare accidentally turned off 1.1.1.1 (I’m simplifying slightly, most notably DNS-over-HTTPS continued to work). Over the course of five or six minutes, traffic dropped to 10% of normal, and stayed there. Somehow, it took another six minutes before an alert fired, at which point they noticed. https://news.ycombinator.com/item?id=44578490 You’d think that for such a company they’d notice if glob…

At my company we will notice that traffic drops significantly for a minute, but thanks to reporting latency we don’t get alerted until a few minutes later. In our business and at our scale, that latency is fine because we aren’t a vital internet service.

edit: I should have followed your link before commenting, because this sentiment is well covered there.

Re: Ask HN: Why are most status pages delayed?

#42

Post by slig »

They lie. Meta ad deliver has been completely fucked for about two months, and they rarely update their status page [1]. When they do, they update with "medium disruptions" hours late and then with "resolved" hours before actually solving. I could understand the "resolved" before because of "web scale" and deploying slowly.

This third partt status page reflects the issues much better [2].

[1]: https://metastatus.com/ads-manager [2]: https://statusgator.com/services/meta

Re: Ask HN: Why are most status pages delayed?

#44

Post by jakevoytko »

mirekrusin wrote: post #39
swiftcoder wrote: post #15

Because for most major sites, updating the status page requires (a significant number of) humans in the loop. Back when I worked at a major cloud provider (which admittedly was >5 years ago), our alarms would go off after ~3-15 minutes of degraded functionality (depending on the sensitivity settings of that specific alarm). At that point the on call gets paged in to investigate and validates that the issue is real (a…

So there is access to "degraded functionality" from start (the "3-15" of "degraded functionality" one) - people are asking why not share THAT then? Nobody cares about internal escalations, if manager is taking shit or not - that's not service status, that's internal dealing with the shit process - it can surface as extra timestamped comments next to service STATUS.

Because the systems are so complex and capable of emergent behavior that you need a human in the loop to truly interpret behavior and impact. Just because an alert is going off doesn't mean that the alert was written properly, or is measuring the correct thing, or the customer is interpreting its meaning correctly, etc.

Re: Ask HN: Why are most status pages delayed?

#45

Post by kachapopopow »

2gremlin181 wrote: post #16
kachapopopow wrote: post #12

because these systems are so big and the people who can validate problems might be asleep at the wheel or be pretty far up the chain and it takes time to reach it. most of the spikes on downdetector are often unrelated to the service, but a 3rd party failure.

IMO if you have an endpoint or service on your status page, you most definitely have an oncall rotation for it. Regarding the second point, your service might be down due to an AWS outage. It's an upstream issue and I fully understand that but I should not have to track things upstream by guessing what cloud provider your use. Where do we draw the line too? What if its not AWS but Hetzner or some other boutique provi…

well usually you have no way to even validate the issue if is due to a bad route and giving out an inaccurate status report is poorly reflected on a pristine:tm: status page. also status updates send out (in some cases) millions of notifications so (global) notifications are only reserved for P0 type issues.

Re: Ask HN: Why are most status pages delayed?

#46

Post by Mojah »

Most companies prefer to fix any downtime before it's noticed, and sharing any details on a status page means admitting something went wrong.

There's plenty of status page solutions that tie in uptime monitoring with status updates, essentially providing a "if we get an alert, anyone can follow along through the status page" for near real-time updates. But, it means showing _all_ users that something went wrong, when maybe only a handful noticed it in the first place.

It's a flawed tactic to try and hide/dismiss any downtime (people will notice), but it's in our human nature to try and hide the bad things?

[1] ie https://ohdear.app/features/status-pages

Re: Ask HN: Why are most status pages delayed?

#47

Post by digitalsushi »

jpalawaga wrote: post #25

It’s not a technical issue, it’s a business one. Those status pages are often linked to contractual SLAs and updating the page tangibly means money lost. So there’s an incentive to only up it when the issue is severe and not quickly remediated. It’s not an engineers tool, it’s a liability tool.

Imagine what you could get away with if you owned the ledger of truth. Any time you made an offence, you could just update that ledger to say the people complaining are wrong, and that's the end of it.

I feel that the tech industry does not have sole ownership of this powerful tool

Re: Ask HN: Why are most status pages delayed?

#48

Post by [deleted] »

swiftcoder wrote: post #40
mirekrusin wrote: post #39

Earlier quoted context omitted.

So there is access to "degraded functionality" from start (the "3-15" of "degraded functionality" one) - people are asking why not share THAT then? Nobody cares about internal escalations, if manager is taking shit or not - that's not service status, that's internal dealing with the shit process - it can surface as extra timestamped comments next to service STATUS.

> why not share THAT then? When you've guaranteed 4 or 5 nines worth of uptime to the customer, every acknowledged outage results in refunds (and potentially being sued over breach of contract)

[deleted]

Re: Ask HN: Why are most status pages delayed?

#50

Post by gwbas1c »

arccy wrote: post #28

Another way to look at it is: you already know the service is down because you can't use it. The status page being manually updated means someone is aware and actually working on fixing it, rather than it being automated and the other side just ignoring it...

That's an extremely insightful comment; I can't believe it's being downvoted.

I would add that I once had to file an FCC complaint when my internet went down. Comcast kept sending me to an AI bot; it appeared that they automated their status page and customer support phone line to the point where I didn't believe that an actual person was aware of the problem and fixing it.

Post reply on HN