Live data from Hacker News

GitHub was down

githubstatus.com

31–40 of 88 posts

Re: GitHub was down

#31

Github folks--this is really getting bad. I find it strange that your leadership will spends weeks of time, and pen hundreds of words about making right the wrongs they created with censorship (see: https://github.blog/2020-11-16-standing-up-for-developers-yo... ), yet there's almost no attention given to these major outages that keep happening for a year now. Where is the acknowledgment of a problem, root-cause anal…

Critical issue: "change master to main across the board"

Re: GitHub was down

#32
post #3

Unfortunately right when we were trying to deploy a hot-fix to production, our CI can't clone the PR to run tests. What do other folks use to avoid this situation? Have a Gitlab instance or similar that you can pull from instead for CI?

You really just need a TCP pathway between your CI and some machine with the git repo on it.

Or take your local copy and use git-fu commands to create a bare repo of it that you can compress and put somewhere like S3. Then download it in CI and checkout from that.

Or just tarball your app source, who cares about git, and do the same (s3, give it a direct path to the asset)

All of this is potentialy useless info though. Hard to say without understanding how your CI works. If all you need is the source code, there are a half dozen ways to get that source into CI without git.

Re: GitHub was down

#33
post #3

Unfortunately right when we were trying to deploy a hot-fix to production, our CI can't clone the PR to run tests. What do other folks use to avoid this situation? Have a Gitlab instance or similar that you can pull from instead for CI?

When I built CI stuff at my previous job there were two remote repos that could be cloned from; Github and a repo on a system on the LAN that the CI's user has ssh access. Which one was used was controlled by a toggle-able environment variable in the CI system.

Re: GitHub was down

#34
I wonder if they track Github Status traffic volume as some sort of meta-indicator? Is it even viable?

I was futzing around with the description for a PR and hitting save wouldn't update it, yet clicking edit would show the text I expected to see.

Suspecting something was up I checked Github Status but it was green across the board. Assuming enough other people hit the same chain of events, could it provide a reliable enough indicator of an issue?

Re: GitHub was down

#35
post #5
post #3

Unfortunately right when we were trying to deploy a hot-fix to production, our CI can't clone the PR to run tests. What do other folks use to avoid this situation? Have a Gitlab instance or similar that you can pull from instead for CI?

Can you make a local build? Our fallback is that someone does the CI/CD steps manually.

Presumably, manually running builds locally is an automatic failsafe option if you have the same people around who originally set the build pipeline up in the first place.

In 2021, basic business continuity plans for software companies should incorporate these sorts of concerns. You should have a published procedure somewhere that a person could follow for producing the final build artifacts of your software on any machine once backups are made available. Situations like these are why I check in 100% of my dependencies to source control as well.

Re: GitHub was down

#36

What's the best practice for high availability (self-hosted?) repositories? Is there a pass-through proxy for git? Or a leader-follower arrangement that is nice, with a proxy server?

Self host GitLab seems like a good approach here.

Re: GitHub was down

#37
post #3

Unfortunately right when we were trying to deploy a hot-fix to production, our CI can't clone the PR to run tests. What do other folks use to avoid this situation? Have a Gitlab instance or similar that you can pull from instead for CI?

Could you just pull the PR from the submitter’s machine? They could serve it via e.g. git http-backend and you could then point the CI there to pull.

Re: GitHub was down

#38

This is becoming a regular occurrence by now.. I wonder if reliability has become less of a priority. As somebody with little to no experience of running things at scale I’m finding myself attributing this to some form of “move fast and break things”.

Given that the change happened in the mid-morning PST (timezone where GitHub HQ and most devs are located), I'm going to bet it's almost certainly something messed up from a regular update or deployment.

Re: GitHub was down

#39

I wonder if they track Github Status traffic volume as some sort of meta-indicator? Is it even viable? I was futzing around with the description for a PR and hitting save wouldn't update it, yet clicking edit would show the text I expected to see. Suspecting something was up I checked Github Status but it was green across the board. Assuming enough other people hit the same chain of events, could it provide a reliabl…

Yes

Re: GitHub was down

#40
Again? Just 11 days ago [0], GitHub Actions had a degraded service and now it is the whole of GitHub. It's becoming a regular thing for them and it really is disappointing.

But I don't know how many times [0] I have to say this but, just get a self-hosted backup rather than 'going all in on GitHub' or 'Centralising everything'.

[0] https://news.ycombinator.com/item?id=26301659

Post reply on HN