Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Incident with Actions and Pages

githubstatus.com

Post reply on HN
Print view View original

401–407 of 407 posts

Re: Incident with Actions and Pages

#401

Post by logifail »

baby_souffle wrote: post #251
logifail wrote: post #219

Earlier quoted context omitted.

> All these monitoring rules are of the format "when 500 errors > baseline for x minutes". Otherwise you'd have monitoring alerts every second. So it is normal for users to already see errors before github officially counts it as an outage. Is it true that official service status pages are updated automatically ?

> it true that official service status pages are updated automatically? Depends. Typically no because there’s an art to crafting the actual message around impact… but sometimes yes it is automated

> Typically no because there’s an art to crafting the actual message around impact

I was thinking more of needing to notify/get sign-off from management...

Re: Incident with Actions and Pages

#402

Post by baby_souffle »

logifail wrote: post #401
baby_souffle wrote: post #251

Earlier quoted context omitted.

> it true that official service status pages are updated automatically? Depends. Typically no because there’s an art to crafting the actual message around impact… but sometimes yes it is automated

> Typically no because there’s an art to crafting the actual message around impact I was thinking more of needing to notify/get sign-off from management...

> I was thinking more of needing to notify/get sign-off from management...

Yeah, that's usually part of it. Precise language matters a TON when you might have some expensive breach-of-SLA terms.

Sometimes the people first responding don't even have the full picture yet and can't fully articulate the impact so they leave it vague.

Re: Incident with Actions and Pages

#403

Post by [deleted] »

filleduchaos wrote: post #313
registeredcorn wrote: post #281

Earlier quoted context omitted.

I'm not arguing with what you're saying, but it does make me wonder: What exactly is the point of the status page, if "it is normal for users to already see errors before GitHub officially counts it as an outage"? Is it more so to have something to link to for managers who aren't using the service have a pretty bar to look at and feel like they are "doing something"? Or is it more of a kind of a way to prevent confir…

There is oddly enough a middle ground between "zero errors whatsoever" and "outage".

[deleted]

Re: Incident with Actions and Pages

#404

Post by noselasd »

neya wrote: post #255
a10c wrote: post #19

My action failed with "Unexpected error fetching GitHub release for tag refs/heads/master: HttpError: Sorry. Your account was suspended" Which certainly made me shit myself, briefly.

It's an eye opener. Think about it - today, it was a mistake. But, what if it really happened? What if you really lost access to all your years of hard work? It's a wake up call. A blessing in disguise to store what matters to you the most locally, backed up offline. Never trust any single provider. Be it MS or Google or Apple. RAID is the way.

I've set up a local gitea now, and configured a few local runners as we test this setup out.

It's a few hours worth of work. Basic git operations and pull requests works fine for us already.

The interesting part will be how much maintenance this will need, and not the least how hard it'll be to port over github actions. We have trivial workflows, but I suspect this conversion will be the painful part.

Re: Incident with Actions and Pages

#405

Post by AndrianV »

hnlmorg wrote: post #383
re-thc wrote: post #372

Earlier quoted context omitted.

> I never mentioned anything about status pages. For context, the parent comment you replied to started with status page. Then are you talking about internal leaks or just guessing? Otherwise besides what's public how do you know they don't know?

It was two comments prior to mine that mentioned status pages. Someone then replied about how it takes a bunch of HTTP response errors for problems to be alerted and thus I commented that application observability would consist of more than just waiting for users to hit errors.

[dead]

Re: Incident with Actions and Pages

#407

Post by vineet005 »

hnlmorg wrote: post #223
dncornholio wrote: post #210

Earlier quoted context omitted.

> If the first they hear of an outage is when user requests start to fail, then that's a failure in their monitoring as well. Isn't that what monitoring actually is? The issue seems to be in their testing, not monitoring.

No, monitoring for HTTP response code is a subset of observability and not one that generally gives you the best insights into which subsystems are misbehaving nor why. There are synthetic tests, where you can generate API request calls or even simulate an entire user journey. These allow you to control the user agent, the payloads, and thus you know anything errors back are actual errors. These are triggered by the…

[flagged]
Post reply on HN