Live data from Hacker News

Incident with Actions and Pages

githubstatus.com

191–200 of 407 posts

Re: Incident with Actions and Pages

#191
post #164

Earlier quoted context omitted.

Yep, they just need to improve their reliability by 2%! https://mrshu.github.io/github-statuses/

This page tells a very different story from GitHub own status page. What is different here?

Part of it is that it considers downtime in any of the services GitHub provides as GitHub being down. So if GitHub had 100 different services, and only one of them was down at any given time (but at least one was always down), then it would show 0% uptime.

Re: Incident with Actions and Pages

#192

Earlier quoted context omitted.

More likely that 'update the Status site' lives a long way down their incident response plan, and they have alarms going off well before that

yeah I mean a company the size of GitHub certainly can’t be expected to have enough staff to walk and chew gum at the same time

If it's like other BigTechs I have worked at, you need director-level signoff and comms team approval to post an outage notice

Re: Incident with Actions and Pages

#194
post #55

Earlier quoted context omitted.

Same. It's weird how I always find out that GitHub is down before GitHub does. Took 15 minutes before it appeared on githubstatus.com

More likely that 'update the Status site' lives a long way down their incident response plan, and they have alarms going off well before that

it should be automatic tho. Probably isn't so they can at least get the one nine on availability

Re: Incident with Actions and Pages

#195
post #14

Insane, we have to come up with contingency plans now for long-duration GitHub outages because we can't safely do deployments. For a service we're paying thousands of $ per year for even though we host runners ourselves...

Sure. Don't use GitHub.

You can now hire me as an overpriced consultant instead of paying Microsoft.

Re: Incident with Actions and Pages

#196
post #83
post #62

Earlier quoted context omitted.

All these monitoring rules are of the format "when 500 errors > baseline for x minutes". Otherwise you'd have monitoring alerts every second. So it is normal for users to already see errors before github officially counts it as an outage.

In a high performance service with good maintenance and upkeep, you page for all 500s. A noisy pager forces the team to fix the 500s. Maybe the Github Actions infrastructure isn't run like that. edit: my oncall rotation notified on all 500s, 24/7, not just rates - https://news.ycombinator.com/item?id=48279262

Yeah, no, nobody runs cloud services like that. At AWS most alarms required failures in 3 consecutive 5 minute periods. Critical things could be on 3 consecutive 1 minute windows - but that alarm starts a 15 minute escalation for the oncall engineer to check in, and they have to validate the issue isn't a false alarm before updating the status page would even be considered

Re: Incident with Actions and Pages

#198
post #28

Earlier quoted context omitted.

Depending on how many thousands of $ per year, it would probably be cheaper and more reliable to self-host GitLab. It's better in terms of organisational structure (you can have one, including access and secret inheritance), and (personal view) Gitlab-CI is better than GitHub Actions because it doesn't push you towards a JavaScript/NPM style dependency hell. And it's actually fairly easy to self-hosted, with options…

Sounds good until you see their cvedetails page

When you own it you can just limit it into vpn-ed company users, that significantly cuts down on the area that can be hit

Re: Incident with Actions and Pages

#199

free service is down again, let's everyone that use the service for free complain again!!! (sorry for the sarcastic comment but i find it crazy how people feel they are entitled when it's free) EDIT: sorry i meant this rant at the one complaining for the free service not for the paid customers (which is unacceptable)

I found it crazy that you haven't discovered that people pay for github

Re: Incident with Actions and Pages

#200

free service is down again, let's everyone that use the service for free complain again!!! (sorry for the sarcastic comment but i find it crazy how people feel they are entitled when it's free) EDIT: sorry i meant this rant at the one complaining for the free service not for the paid customers (which is unacceptable)

If you take a bit of a closer look, github.com has a "pricing" page

https://github.com/pricing

Post reply on HN