GitHub was down • Hacker News Board

Re: GitHub was down

#21

Post by fweespeech » Fri, Mar 12, 2021, 6:47 PM UTC

cs-szazz wrote: post #3

Unfortunately right when we were trying to deploy a hot-fix to production, our CI can't clone the PR to run tests. What do other folks use to avoid this situation? Have a Gitlab instance or similar that you can pull from instead for CI?

> Have a Gitlab instance or similar that you can pull from instead for CI?

Gitlab, mirrored repo basically.

Re: GitHub was down

#22

Post by drstewart » Fri, Mar 12, 2021, 6:47 PM UTC

justaguy88 wrote: post #8

What's the best practice for high availability (self-hosted?) repositories? Is there a pass-through proxy for git? Or a leader-follower arrangement that is nice, with a proxy server?

Best practice for me is to hope this happens on Friday afternoon and then take a half day... so all according to plan!

Re: GitHub was down

#23

Post by capableweb » Fri, Mar 12, 2021, 6:49 PM UTC

cs-szazz wrote: post #3

Unfortunately right when we were trying to deploy a hot-fix to production, our CI can't clone the PR to run tests. What do other folks use to avoid this situation? Have a Gitlab instance or similar that you can pull from instead for CI?

It's surprisingly easy, depending on your scale/scope of course. But in general, I've managed to build CI/CD pipelines that are tolerant of GitHub (or any service) failures by following these steps:

1. Use as little of the configuration language provided by the CI as possible (prefer shellscripts that you call in CI instead of having each step in a YAML config for example)

2. Make sure static content is in a Git repository (same or different) that is also available on multiple SCM systems (I usually use GitHub + GitLab mirroring + a private VPS that also mirrors GitHub)

3. Have a bastion host for doing updates, make CI push changes via bastion host and have at least four devs (if you're at that scale, otherwise you just) with access to it, requiring multisig of 2 of them to access

Now when the service goes down, you just need 2 developers to sign the login for the bastion host, then manually run the shellscript locally to push your update. You'll always be able to update now :)

Re: GitHub was down

#24

Post by leemac » Fri, Mar 12, 2021, 6:49 PM UTC

Sent in a commit to fix a few PR suggestions. It went through, but nowhere to be found on the PR. Guess I'll have to wait.

Re: GitHub was down

#25

Post by smallnamespace » Fri, Mar 12, 2021, 6:49 PM UTC

justaguy88 wrote: post #8

What's the best practice for high availability (self-hosted?) repositories? Is there a pass-through proxy for git? Or a leader-follower arrangement that is nice, with a proxy server?

If you're already using a cloud provider, they [1] all [2] have [3] private git repo services.

You can set up a cronjob to sync them, or some have built-in config to do the mirroring [4].

I used Google's mirroring option before. It was fine, but we never had to use it (local copies were sufficient when GH was slow one day).

[1] https://cloud.google.com/source-repositories

[2] https://aws.amazon.com/codecommit/

[3] https://azure.microsoft.com/en-us/services/devops/repos/

[4] https://cloud.google.com/source-repositories/docs/mirroring-...

Re: GitHub was down

#26

Post by suspecthorse » Fri, Mar 12, 2021, 6:49 PM UTC

I started building Multiverse because of problems like this. Ironically it’s hosted on GitHub. Check it out if you are interested in decentralized VCS and code hosting.

https://github.com/multiverse-vcs/go-multiverse

Re: GitHub was down

#27

Post by theptip » Fri, Mar 12, 2021, 6:49 PM UTC

cs-szazz wrote: post #3

Unfortunately right when we were trying to deploy a hot-fix to production, our CI can't clone the PR to run tests. What do other folks use to avoid this situation? Have a Gitlab instance or similar that you can pull from instead for CI?

Ideally your CI/CD is just calling Make/Python/Whatever scripts that are one shot actions. You should be able to run the same action locally from a clean git repo (assuming you have the right permissions).

The anti-pattern to watch out for is long, complex scripts that live in your CI system’s config file. These are hard to test and replicate when you need to.

Re: GitHub was down

#28

Post by qbasic_forever » Fri, Mar 12, 2021, 6:49 PM UTC

Github folks--this is really getting bad. I find it strange that your leadership will spends weeks of time, and pen hundreds of words about making right the wrongs they created with censorship (see: https://github.blog/2020-11-16-standing-up-for-developers-yo...), yet there's almost no attention given to these major outages that keep happening for a year now.

Where is the acknowledgment of a problem, root-cause analysis, and followup for new practices and engineering to prevent issues? Who is responsible for these issues and what are they doing to make it right? What positions are you hiring for _right now_ to get to work making your service reliable?

Re: GitHub was down

#29

Post by whoisjohnkid » Fri, Mar 12, 2021, 6:50 PM UTC

turbonoobie wrote: post #17

This is becoming a regular occurrence by now.. I wonder if reliability has become less of a priority. As somebody with little to no experience of running things at scale I’m finding myself attributing this to some form of “move fast and break things”.

good point; I think they might be deploying big features on fridays? as of late I've noticed most of the issues I have seen happen on Fridays.

Re: GitHub was down

#30

Post by mfer » Fri, Mar 12, 2021, 6:50 PM UTC

Running a highly available service at this scale is hard. Especially when the service is ripe for dos and attacks.

With that out of the way... GH has had a lot of issues in recent months. More than the past. I would hope those things are on a road to being fixed.