Github folks--this is really getting bad. I find it strange that your leadership will spends weeks of time, and pen hundreds of words about making right the wrongs they created with censorship (see: https://github.blog/2020-11-16-standing-up-for-developers-yo... ), yet there's almost no attention given to these major outages that keep happening for a year now. Where is the acknowledgment of a problem, root-cause anal…
GitHub was down
31–40 of 88 posts
Re: GitHub was down
#32Unfortunately right when we were trying to deploy a hot-fix to production, our CI can't clone the PR to run tests. What do other folks use to avoid this situation? Have a Gitlab instance or similar that you can pull from instead for CI?
Or take your local copy and use git-fu commands to create a bare repo of it that you can compress and put somewhere like S3. Then download it in CI and checkout from that.
Or just tarball your app source, who cares about git, and do the same (s3, give it a direct path to the asset)
All of this is potentialy useless info though. Hard to say without understanding how your CI works. If all you need is the source code, there are a half dozen ways to get that source into CI without git.
Re: GitHub was down
#33Unfortunately right when we were trying to deploy a hot-fix to production, our CI can't clone the PR to run tests. What do other folks use to avoid this situation? Have a Gitlab instance or similar that you can pull from instead for CI?
Re: GitHub was down
#34I was futzing around with the description for a PR and hitting save wouldn't update it, yet clicking edit would show the text I expected to see.
Suspecting something was up I checked Github Status but it was green across the board. Assuming enough other people hit the same chain of events, could it provide a reliable enough indicator of an issue?
Re: GitHub was down
#35Unfortunately right when we were trying to deploy a hot-fix to production, our CI can't clone the PR to run tests. What do other folks use to avoid this situation? Have a Gitlab instance or similar that you can pull from instead for CI?
Can you make a local build? Our fallback is that someone does the CI/CD steps manually.
In 2021, basic business continuity plans for software companies should incorporate these sorts of concerns. You should have a published procedure somewhere that a person could follow for producing the final build artifacts of your software on any machine once backups are made available. Situations like these are why I check in 100% of my dependencies to source control as well.
Re: GitHub was down
#36What's the best practice for high availability (self-hosted?) repositories? Is there a pass-through proxy for git? Or a leader-follower arrangement that is nice, with a proxy server?
Re: GitHub was down
#37Unfortunately right when we were trying to deploy a hot-fix to production, our CI can't clone the PR to run tests. What do other folks use to avoid this situation? Have a Gitlab instance or similar that you can pull from instead for CI?
Re: GitHub was down
#38This is becoming a regular occurrence by now.. I wonder if reliability has become less of a priority. As somebody with little to no experience of running things at scale I’m finding myself attributing this to some form of “move fast and break things”.
Re: GitHub was down
#39I wonder if they track Github Status traffic volume as some sort of meta-indicator? Is it even viable? I was futzing around with the description for a PR and hitting save wouldn't update it, yet clicking edit would show the text I expected to see. Suspecting something was up I checked Github Status but it was green across the board. Assuming enough other people hit the same chain of events, could it provide a reliabl…
Re: GitHub was down
#40But I don't know how many times [0] I have to say this but, just get a self-hosted backup rather than 'going all in on GitHub' or 'Centralising everything'.