Created https://gitbackup.org for this very reason.
GitHub is degraded/down
51–60 of 171 posts
Re: GitHub is degraded/down
#52Earlier quoted context omitted.
It might also be covid related. People are working from home, people responsible for system upkeep might not be immediately responsive, more demand on the servers for whatever reason, etc.
I would agree that the coronavirus could be a factor here. At the same time, I've been noticing issues since probably December or January (before the coronavirus started being a real problem), which makes it seem like maybe there are multiple issues. Of course, I'm not actually internal to Microsoft or Github, so I have no idea and it's all opaque to me.
Alternative explanation - they've been deploying big new features with some regular cadence lately. New features carry risk and I think we're seeing that.
Re: GitHub is degraded/down
#53I don't think it's just GitHub. I noticed everything seemed slow but thought it was just my ISP. Turns out from looking at downdetector.com there seems to be a large spike in reported problems across many sites and providers, all occurring at the same time.
Maybe the cloud providers are running out of capacity?
Re: GitHub is degraded/down
#54Created https://gitbackup.org for this very reason.
Where do you store all that data?
Re: GitHub is degraded/down
#55Earlier quoted context omitted.
Github is as much or more a collaboration tool as a Git upstream. The Git part is easy to distribute. The collaboration tool, not so much. And to anticipate the likely next argument - no, mailing lists are not a better tool for collaboration. They are distributed, sure. In so saying, I have exhausted the list of their virtues.
The "we deploy production directly from Github" pathology is the one I'm talking about. Not being able to "collaborate" for a brief outage is fine. Not being able to deploy code isn't.
You gotta CI/CD from somewhere. Whatever that somewhere is, it can go down.
You can, of course, override your CI/CD and do a manual deploy. It's not a matter of "can." It's a matter of not fully understanding all the checks the CI/CD system does to the code, and all the build steps for prod builds, and therefore not having the confidence to deploy to prod without CI/CD holding your hand. (Which I don't at-all blame devs in bigcorps for not knowing; release management is its whole own thing, and division of labor means that at sufficient scale it doesn't make sense to learn about it.)
Re: GitHub is degraded/down
#56Is it just me, or has Github's quality of service been continually degrading over the past several months? What is going on internally? Is this because of the Microsoft acquisition? Increased usage? An internal transition to Azure? ...is it time to move away from Github?
For hosting your own repos AWS CodeCommit works very well.
Re: GitHub is degraded/down
#57Today marks the 3rd time I've broached the topic w/ management of getting the self-host enterprise option... Compared to (public) GitHub's problems this year so far, our AWS EC2 instances are orders of magnitude more reliable.
Sure, the internet can still go down, but my VPN into us-east-1 from Texas has been unbroken for weeks now.
At this point I'd almost prefer to pull it all in-house and manage it myself so the entire team doesn't have to lose a whole day of productivity over all this. I am so glad we moved away from using GH Actions for builds because we would be absolutely hosed right now on supporting our customers.
Re: GitHub is degraded/down
#58I don't think it's just GitHub. I noticed everything seemed slow but thought it was just my ISP. Turns out from looking at downdetector.com there seems to be a large spike in reported problems across many sites and providers, all occurring at the same time.
I thought this at first too but downdetector sorts all the problem sites to the top which makes it look like the internet is melting when you first look at it. It actually seems reasonable for this many sites to be having issues at any one given time.
Re: GitHub is degraded/down
#59Earlier quoted context omitted.
Github is as much or more a collaboration tool as a Git upstream. The Git part is easy to distribute. The collaboration tool, not so much. And to anticipate the likely next argument - no, mailing lists are not a better tool for collaboration. They are distributed, sure. In so saying, I have exhausted the list of their virtues.
The "we deploy production directly from Github" pathology is the one I'm talking about. Not being able to "collaborate" for a brief outage is fine. Not being able to deploy code isn't.
All those deployments typically rely on:
1. Scripts available in the git repository.
2. Parameters that are either stored (possibly encrypted) in a filesystem, on a parameter server (like AWS SSM key manager), or in 1Password/LastPass.
It's only a pathology if a team couldn't, as a worst case, coordinate a deployment from a single machine over the phone.
Re: GitHub is degraded/down
#60Created https://gitbackup.org for this very reason.
Where do you store all that data?
Should only be 2-3 PB. Storing it all on https://tardigrade.io