They should have used Kardinal: https://github.com/kurtosis-tech/kardinal
GitHub was down
251–260 of 288 posts
Re: GitHub was down
#252Earlier quoted context omitted.
This is intentional. It's mostly a matter of discussing how to communicate it publicly and when to flip the switch to start the SLA timer. Also coordinating incident response during a huge outage is always challenging.
That it may be but there’s no excuse. Declare an incident first, investigate later. Cheating SLAs by delaying the incident is a good way to erode trust within and without.
> Declare an incident first, investigate later.
If that would be the best way to deal with it- why is literally no one doing it this way and what does that tell you?
Re: GitHub was down
#253This is weird. I've been using Github all night (in France) and didn't notice anything was wrong. Was the outage in North America?
Re: GitHub was down
#254Earlier quoted context omitted.
That it may be but there’s no excuse. Declare an incident first, investigate later. Cheating SLAs by delaying the incident is a good way to erode trust within and without.
> Declare an incident first, investigate later. If that would be the best way to deal with it- why is literally no one doing it this way and what does that tell you?
because it involves admitting that you messed up which companies are often disensentivized to do
Re: GitHub was down
#255Earlier quoted context omitted.
I get the angry unicorn page "No server is currently available to service your request. Sorry about that. Please try refreshing and contact us if the problem persists. Contact Support — GitHub Status — @githubstatus" with that last link going to https://x.com/githubstatus showing "GitHub Status Oct 22, 2018 Everything operating normally."
The era of Twitter/X status pages needs to come to an end given how unusable it is if you aren't logged in.
Making logins required to view twitter was the ultimate bed shitting move. The whole point of twitter was to be a broadcast medium. Tweets were viewable without following or logging in. There is a huge vacuum in that space now.
Re: GitHub was down
#256Could it have been brought down intentionally? Related to this? https://www.bleepingcomputer.com/news/security/github-action...
Seems like it was a config change that cause it. They reverted it reality quickly.
Re: GitHub was down
#257Earlier quoted context omitted.
They're already mirrored on my hard drive. That's how git works.
Github is more than a remote host for git repositories. It's become one of the major CDNs for software distributions. Github Pages host a majority of static sites that developer use. You won't be able to use Cargo, Nix, Scoop and other package managers right now because their registries have a critical dependency hosted on Github. This is not to mention all the projects that rely on Github for project management, dev…
- Champion a hard to use VCS which to its credit is distributed
- Make everyone dependent on all the centralized features of your software to use Git[1][2]
- Now you have a de facto centralized, hard to use VCS with thousands of SO questions like “my code won’t commit to the GitHub”
- Every time you go down a hundreds-of-comments post is posted on HN
How to get bought for a ton of cash by a tech mega corporation.
[1] Of course an exaggeration. Everyone can use it in a distributed way or mirror. The problem occurs when you’re on a team and everyone else doesn’t know how to.
[2] I’m pretty sure that even the contributors to the Git project rely on the GitHub CI since they can’t run all tests locally.
Re: GitHub was down
#258Earlier quoted context omitted.
At Github's scale, why wouldn't they put a ping monitor from every continent at least? Then, you would show the status based on the continent.
Where on the continent? GitHub is undoubtedly doing blackbox testing internally and has multiple such monitors but that's not going to capture every customer's route to them, leading to the same problem - customers experience GitHub being down, despite monitoring saying it's mostly up. Thus the impass. Even doing whitebox testing, where you know the internals and can this place sensors intelligently, even just for in…
> 100% fidelity is impossible
I don't think anybody asked for 100% fidelity. We are talking about a complete outage that affected at least North America and Europe. If the status page shows green in such a case, its fidelity is around 50%. People expect better from GitHub.
Re: GitHub was down
#259https://www.githubstatus.com/ reports no problems, but it's clearly down for a lot of people (including me).
It is kinda amazing how consistently status pages show everything fine during a total outage. It's not that hard to connect a status page to end-to-end monitoring statistics...
They say you shouldn't host status pages on the same infrastructure that it is monitoring, but in a way that makes it much more accurate and responsive in outages!
Re: GitHub was down
#260I've never seen an outage this big. Even the homepage doesn't load. We've had recurrent issues with Actions not running, but this seems a lot bigger. The status page says all is well, though: https://www.githubstatus.com/ . Hilarious.
FWIW, our self-hosted Gitea instance has not had a single second of unplanned downtime in five years we've been running it. And there wasn't much _planned_ downtime because it's really easy to upgrade (pull a new image and recreate the container — takes out the instance for maybe 15 seconds late at night), and full backups are handled live thanks to zfs.
Migration to a new host takes another 15 seconds thanks to both zfs and containers.
I don't know how many GitHub downtime reports I've seen during that time, we're probably into high dozens by now.