As long as the same problem doesn't occur again, I have no hard feelings against them.
GitHub: October 21 Incident Report
31–40 of 119 posts
Re: GitHub: October 21 Incident Report
#32Earlier quoted context omitted.
Lots of users. Also, a lot of package managers rely on hosting platforms like GitHub to host their packages, so if Github breaks, a lot of CI processes around the world break.
Which is kind of ridiculous. If your CI breaks because GitHub is down, it means it's not caching dependencies locally, but keeps re-downloading them every time it runs (e.g. every commit), generating tons of waste and unnecessary load on the hosting service. Or, to put it bluntly, if your CI works like this, it's contributing to climate change.
I think you are wrong. Our CI infra caches all dependencies, but it depends on github for new internal code pushes (kind of the point). If github is not sending events, CI doesnt kick.
Youre ignoring half of the problem. If you dont receive events from github because they are down, your CI doesnt work either -- dependency caching doesnt matter at that point.
Re: GitHub: October 21 Incident Report
#33Lets do a quick back of the envelope calculation:
Github reports 28,337,706 users by 2018-06-05 [1]. Lets assume 50% of these are active. Lets also assume that, due to the unavailability of GH, around 2 usable hours per developer are lost. Another assumption is that each developer contributes around 50 US$ per hour.
This means, this outage has cost us users: (28337706 * .5 * 2 * 50) = 1.351 billion US$.
Perhaps not use MySQL for such critical systems?
Re: GitHub: October 21 Incident Report
#34Earlier quoted context omitted.
Which is kind of ridiculous. If your CI breaks because GitHub is down, it means it's not caching dependencies locally, but keeps re-downloading them every time it runs (e.g. every commit), generating tons of waste and unnecessary load on the hosting service. Or, to put it bluntly, if your CI works like this, it's contributing to climate change.
I think you are wrong. Our CI infra caches all dependencies, but it depends on github for new internal code pushes (kind of the point). If github is not sending events, CI doesnt kick. Youre ignoring half of the problem. If you dont receive events from github because they are down, your CI doesnt work either -- dependency caching doesnt matter at that point.
That's assuming you're putting your own organization's code on GitHub. Then of course if GitHub doesn't work, neither does the CI that's hooked to it. This is a separate topic.
Re: GitHub: October 21 Incident Report
#35I don't understand why code hosting platforms like GitHub, GitLab or BitBucket have so many issues regularly. Is there anything special about it?
No, that's what a typical single organization with a typical RDBMS-centric non-resilient architecture can provide. But of course it's very hard for organizations at certain sizes to do it better, it's something they have to start with.
Re: GitHub: October 21 Incident Report
#36Lets do a quick back of the envelope calculation: Github reports 28,337,706 users by 2018-06-05 [1]. Lets assume 50% of these are active. Lets also assume that, due to the unavailability of GH, around 2 usable hours per developer are lost. Another assumption is that each developer contributes around 50 US$ per hour. This means, this outage has cost us users: (28337706 * .5 * 2 * 50) = 1.351 billion US$. Perhaps not u…
That's a vast exaggeration. You're assuming every single one of GitHub's active users: 1) was active during the exact incident time and 2) is a business user earning US wages.
Re: GitHub: October 21 Incident Report
#37Lets do a quick back of the envelope calculation: Github reports 28,337,706 users by 2018-06-05 [1]. Lets assume 50% of these are active. Lets also assume that, due to the unavailability of GH, around 2 usable hours per developer are lost. Another assumption is that each developer contributes around 50 US$ per hour. This means, this outage has cost us users: (28337706 * .5 * 2 * 50) = 1.351 billion US$. Perhaps not u…
Average 50 percent active (as in daily usage)? 2 hours lost?
I'd estimate a couple of order of magnitudes less...
Re: GitHub: October 21 Incident Report
#38Yeah, we're still facing issues with, erm, Github issues. Also, while they haven't updated this blog post for a while, their status page has been very up-to-date and informative: https://status.github.com/messages
> very up-to-date and informative Is that satire? It said 2-hour ETA 5 hours ago and the last update was over two hours ago.
The latest message appears to be:
"We are validating the consistency of information across all data stores. Webhooks and Pages builds remain paused."
Which is a bit scary. Half my requests appear to hit some storage which is still many hours behind. They should be seeing that...
Re: GitHub: October 21 Incident Report
#39Lets do a quick back of the envelope calculation: Github reports 28,337,706 users by 2018-06-05 [1]. Lets assume 50% of these are active. Lets also assume that, due to the unavailability of GH, around 2 usable hours per developer are lost. Another assumption is that each developer contributes around 50 US$ per hour. This means, this outage has cost us users: (28337706 * .5 * 2 * 50) = 1.351 billion US$. Perhaps not u…
Average 50 percent active (as in daily usage)? 2 hours lost? I'd estimate a couple of order of magnitudes less...
I don't think 2 hours is an exaggeration for an active user. The 50% might be too high, I agree.
Re: GitHub: October 21 Incident Report
#40Lets do a quick back of the envelope calculation: Github reports 28,337,706 users by 2018-06-05 [1]. Lets assume 50% of these are active. Lets also assume that, due to the unavailability of GH, around 2 usable hours per developer are lost. Another assumption is that each developer contributes around 50 US$ per hour. This means, this outage has cost us users: (28337706 * .5 * 2 * 50) = 1.351 billion US$. Perhaps not u…
That's a vast exaggeration. You're assuming every single one of GitHub's active users: 1) was active during the exact incident time and 2) is a business user earning US wages.
How would you do the calculation of the costs of this outage? Don't forget that the 50 US$ is lost added value, not wages.