Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Incident with Actions and Pages

githubstatus.com

Post reply on HN
Print view View original

81–90 of 407 posts

Re: Incident with Actions and Pages

#82

Post by re-thc »

drcongo wrote: post #55
a10c wrote: post #19

My action failed with "Unexpected error fetching GitHub release for tag refs/heads/master: HttpError: Sorry. Your account was suspended" Which certainly made me shit myself, briefly.

Same. It's weird how I always find out that GitHub is down before GitHub does. Took 15 minutes before it appeared on githubstatus.com

> It's weird how I always find out that GitHub is down before GitHub does

No, it's not. Official updates = potential SLA penalties. Always requires approval.

Re: Incident with Actions and Pages

#83

Post by echelon »

jaapz wrote: post #62
drcongo wrote: post #55

Earlier quoted context omitted.

Same. It's weird how I always find out that GitHub is down before GitHub does. Took 15 minutes before it appeared on githubstatus.com

All these monitoring rules are of the format "when 500 errors > baseline for x minutes". Otherwise you'd have monitoring alerts every second. So it is normal for users to already see errors before github officially counts it as an outage.

In a high performance service with good maintenance and upkeep, you page for all 500s. A noisy pager forces the team to fix the 500s.

Maybe the Github Actions infrastructure isn't run like that.

edit: my oncall rotation notified on all 500s, 24/7, not just rates - https://news.ycombinator.com/item?id=48279262

Re: Incident with Actions and Pages

#84

Post by mystifyingpoi »

decodebytes wrote: post #29
bouk wrote: post #14

Insane, we have to come up with contingency plans now for long-duration GitHub outages because we can't safely do deployments. For a service we're paying thousands of $ per year for even though we host runners ourselves...

Same thoughts - we use an action to ship to production, its builds an image, pushes it to ECS which triggers a deployment. We can't be blocked here. Seems silly what we settled on this, but for a long time GitHub had been reliable enough for many years, but things are sliding down the pan as of late.

Sounds like a very easy process to rewrite in bash/python and have it on hand if needed.

Re: Incident with Actions and Pages

#85

Post by re-thc »

bouk wrote: post #14

Insane, we have to come up with contingency plans now for long-duration GitHub outages because we can't safely do deployments. For a service we're paying thousands of $ per year for even though we host runners ourselves...

> For a service we're paying thousands of $ per year for even though we host runners ourselves...

Wait until you charge you for self-hosting runners.

Oh wait. They already tried.

Post reply on HN