GitHub Is Down • Hacker News Board

Re: GitHub Is Down

#41

Post by gadrev » Thu, Apr 02, 2020, 8:58 PM UTC

Just managed to push something, but the web interface is still down.

EDIT: (21:00 UTC ) Just got in :D

Re: GitHub Is Down

#42

Post by thanksforfish » Thu, Apr 02, 2020, 9:00 PM UTC

corndoge wrote: post #18

HugoDias wrote: post #12

That's great. 2 minutes before it went down a pipeline workflow ran in Heroku deploying a bug to my app. Now I can't deploy the hotfix since it depends on Github and I'm not able to revert to the last successful build on the Heroku interface. Any ideas?

Remove spiderweb dependencies from your deployment pipeline?

This is a good point, but it could be worded better.

One of the things a site reliability engineer should think about is how well the site can be operated when dependencies have issue. After an incident like this, even if you were able to recover, it's worth thinking about how things could have gone better.

In the past I had a painful experience with one application I was supporting that needed to install NPM packages on deployment. We couldn't successfully deploy (or scale up) for the duration of that outage. In that case we realized it was safer to switch to server images with all assets pre-installed and an NPM cache to give the build a better chance of succeeding. The next NPM outage we only noticed after the fact :)

Not certain how this particular deployment pipeline is failing due to the GitHub outage, but a post-mortem to discuss may be helpful and protect against future issues.

Re: GitHub Is Down

#43

Post by jjordan » Thu, Apr 02, 2020, 9:00 PM UTC

Well, I guess I'm done working for the day.

Re: GitHub Is Down

#44

Post by ogre_codes » Thu, Apr 02, 2020, 9:01 PM UTC

amarshall wrote: post #29

ogre_codes wrote: post #21

Apparently the earlier thread about Microsoft's cloud stability was a bit prescient.

Has GitHub moved their infrastructure to Azure? Last I recall they ran mostly on their own servers.

I'm not sure. My post was mostly in jest. But being at Microsoft clearly hasn't helped.

Re: GitHub Is Down

#45

Post by keeganwitt » Thu, Apr 02, 2020, 9:01 PM UTC

Hehe, somebody modified the XKCD "Compiling" comic.

https://twitter.com/PonteIneptique/status/783190337602846720

Re: GitHub Is Down

#46

Post by spir » Thu, Apr 02, 2020, 9:02 PM UTC

PSA: I just received a github phishing request that was not detected by Gmail. I almost fell for it.

Phishing email looks like a "review your suspicious activity" alert, but the alert is the suspicious activity.

https://imgur.com/a/zdtWmuN

Re: GitHub Is Down

#47

Post by intsunny » Thu, Apr 02, 2020, 9:02 PM UTC

UTC for status updates?? Not PST/PDT??

Amazing!

Re: GitHub Is Down

#48

Post by ralphael » Thu, Apr 02, 2020, 9:02 PM UTC

...its back now for me.

Re: GitHub Is Down

#49

Post by enraged_camel » Thu, Apr 02, 2020, 9:02 PM UTC

jakejarvis wrote: post #28

Even our static assets from raw.githubusercontent.com are throwing 500s. Whatever it is, it sounds like a pretty widespread failure...

Yeah, can confirm. I'm working on some visualizations for covid-19 where I pull data from raw.githubusercontent.com and I'm getting 500s.

Re: GitHub Is Down

#50

Post by thanksforfish » Thu, Apr 02, 2020, 9:03 PM UTC

gramakri wrote: post #37

We use a self-hosted GitLab ourselves and I can honestly say we have more up time than GitHub. But 9/10 every project we use is on GitHub and our work is stalled. Centralization will be our downfall :(

You may be able to mirror those to your GitLab and then do all your work from the mirror.

https://docs.gitlab.com/ee/user/project/repository/repositor...

Congrats on the self-hosted uptime :)