Live data from Hacker News

GitHub: October 21 Incident Report

blog.github.com

41–50 of 119 posts

Re: GitHub: October 21 Incident Report

#41
post #39
post #37

Earlier quoted context omitted.

Average 50 percent active (as in daily usage)? 2 hours lost? I'd estimate a couple of order of magnitudes less...

I don't think 2 hours is an exaggeration for an active user. The 50% might be too high, I agree.

2 hours lost? Maybe 2 hours affected, but then just don't push/fetch during that time. Very few would have to stop and wait.

Average lost I'd guess to 10 minutes (most of it being "huh, wonder what's up with github")

Re: GitHub: October 21 Incident Report

#42
As I rearrange today's todo list, I recall wishing that I'd used the Microsoft purchase event to encourage folks to increase their familiarity with gitlab. So I now note:

https://gitlab.com/explore/projects is a live feed of project activity, suitable for code surfing. It also sorts by stars and trending.

Re: GitHub: October 21 Incident Report

#44
post #10
post #6

I don't understand why code hosting platforms like GitHub, GitLab or BitBucket have so many issues regularly. Is there anything special about it?

Define regularly. I can recall only 2 incidents this year. I think that's not too bad considering the level of traffic they have to contend with.

Considering this is going on for several hours now, their SLA is down to at least 99.9 and going down by the hour. Their business SLA is 99.95% (though I have no idea what it refers to), so it's quite possible that they are in breach.

Still not bad, but 2 incidents like this a year, is usually considered unacceptable for infrastructure service providers.

Re: GitHub: October 21 Incident Report

#45
post #33

Lets do a quick back of the envelope calculation: Github reports 28,337,706 users by 2018-06-05 [1]. Lets assume 50% of these are active. Lets also assume that, due to the unavailability of GH, around 2 usable hours per developer are lost. Another assumption is that each developer contributes around 50 US$ per hour. This means, this outage has cost us users: (28337706 * .5 * 2 * 50) = 1.351 billion US$. Perhaps not u…

The assumption that developer time is lost when Github is unavailable is wrong. The whole idea of Git being a distributed VCS that it does not require any connection to the main server (i.e. Github) to work with a local copy of the repository. If Github is down, I can still do my work locally and then push changes to Github when it's back online. The only case when I may get blocked, is when I need to fetch project dependencies hosted on Github to do the initial build of my project, which doesn't happen very often.

Re: GitHub: October 21 Incident Report

#46
post #10

Earlier quoted context omitted.

Define regularly. I can recall only 2 incidents this year. I think that's not too bad considering the level of traffic they have to contend with.

Considering this is going on for several hours now, their SLA is down to at least 99.9 and going down by the hour. Their business SLA is 99.95% (though I have no idea what it refers to), so it's quite possible that they are in breach. Still not bad, but 2 incidents like this a year, is usually considered unacceptable for infrastructure service providers.

I wonder how they define their SLA though. If only some of the features are down, does it impact the SLA?

Re: GitHub: October 21 Incident Report

#47

Earlier quoted context omitted.

Lots of users. Also, a lot of package managers rely on hosting platforms like GitHub to host their packages, so if Github breaks, a lot of CI processes around the world break.

Which is kind of ridiculous. If your CI breaks because GitHub is down, it means it's not caching dependencies locally, but keeps re-downloading them every time it runs (e.g. every commit), generating tons of waste and unnecessary load on the hosting service. Or, to put it bluntly, if your CI works like this, it's contributing to climate change.

Or it's something using something like cargo (rust's package manager) - and checking if any dependencies have a newer version by checking the package registry (which is stored on github for no apparent reason).

Re: GitHub: October 21 Incident Report

#48
post #43

Don't try creating a new repo, it will create part of the meta data but not allow you to see or use the repo and the repo name gets taken.

I created a new repository at about 9 UTC, and it started to work some of the time at 12 UTC — I've pushed the code from my computer, and pulled it from elsewhere.

However, it's still intermittently failing as of 14 UTC, so I haven't managed a Maven release build yet.

Re: GitHub: October 21 Incident Report

#49
I saw some comments on reddit which highlighted a pretty serious problem - many orgs rely on github as a fully integrated CD platform, with everything from code hosting, to running CI hooks, to pushing to staging or prod.

It seems very unwise to have essentially your whole deployment process manager in the hands of an entity which you don't only have no control over, but which has regularly been targeted in attacks by nation-state-level actors because of their role as a code hosting platform.

EDIT: GH hasn't been targeted regularly, but it has been so historically, so this is a plausible thing which might happen again.

Re: GitHub: October 21 Incident Report

#50
post #30
post #10

Earlier quoted context omitted.

Define regularly. I can recall only 2 incidents this year. I think that's not too bad considering the level of traffic they have to contend with.

> Define regularly More often than what is considered a standard 99.99% uptime SLA? (about an hour per year.) You seem to be making it out like a couple of days a year of lost [1] developer productivity is no big deal. That said, these things happen and you should probably check your workflows if you're all that blocked by GitHub being down.

> More often than what is considered a standard 99.99% uptime SLA?

GitHub SLA is 99.95% and apparently exclusive to Business Cloud customers[1].

[1] https://github.com/pricing

Post reply on HN