Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

GitHub was down

github.com

Post reply on HN
Print view View original

241–250 of 288 posts

Re: GitHub was down

#241

Post by colimbarna »

Joe_Cool wrote: post #217
cropcirclbureau wrote: post #188

Earlier quoted context omitted.

Github is more than a remote host for git repositories. It's become one of the major CDNs for software distributions. Github Pages host a majority of static sites that developer use. You won't be able to use Cargo, Nix, Scoop and other package managers right now because their registries have a critical dependency hosted on Github. This is not to mention all the projects that rely on Github for project management, dev…

We installed a private GitLab instance on our own servers exactly out of fear that Github might suddenly alter the deal or just cease operations. Pretty happy with our decision so far.

Do you mean you switched to self-managed GitLab, or you have a self-managed GitLab that you keep around as a backup plan?

Re: GitHub was down

#242

Post by thiagocsf »

x86a wrote: post #81
purkka wrote: post #69

Earlier quoted context omitted.

I have to wonder how a company at the scale of GitHub can be so bad at keeping track of their status. Now 4 out of 10 services are marked as "Incident", yet most of the others are also completely dead.

This is intentional. It's mostly a matter of discussing how to communicate it publicly and when to flip the switch to start the SLA timer. Also coordinating incident response during a huge outage is always challenging.

That it may be but there’s no excuse.

Declare an incident first, investigate later.

Cheating SLAs by delaying the incident is a good way to erode trust within and without.

Re: GitHub was down

#243

Post by paledot »

bitbasher wrote: post #130
red_Seashell_32 wrote: post #108

Earlier quoted context omitted.

Wait. You use github pages for something or actually work on it?

I use it for something. I had a github page that was public, but it was made private and the DNS config was removed. Fast forward to today. I made the private repo public again and forced a deploy of the page without making a new commit. It said the DNS config was incomplete, so I tweaked it and hit "check again" and github went down. Probably unrelated, but the timing was spooky.

Your domain isn't `null.example.com` or something, is it?

Re: GitHub was down

#245

Post by katzinsky »

dlahoda wrote: post #33

I see more and more people use less Github, but some other git solutions. I am afraid to think what to do when GitHub is down for hours (need to learn maillists?). Another reason is that MS may be in phase when it will ask to pay for using GitHub just for reads (rate limiter).

Git without github is pretty much the same as with it. It's just PRs that are different.

Re: GitHub was down

#246

Post by bill_lumbergh »

2YwaZHXV wrote: post #226
esrauch wrote: post #175

Earlier quoted context omitted.

They probably have a reverse proxy in front of all their http endpoints and that is still up and able to show the unicorn if the backends aren't responsive. The static content on the error page might also be on akami or cloudflare side.

makes sense, thanks. the images on the page are all just base64 encoded right into the html

https://github.blog/news-insights/the-library/unicorn/

  Unicorn has a slightly different architecture.
  Instead of the nginx => haproxy => mongrel cluster setup
  you end up with something like: nginx => shared socket => unicorn worker pools
  
  When the Unicorn master starts, it loads our app into memory. As soon as it’s ready to serve requests it forks 16 workers. Those workers then select() on the socket, only serving requests they’re capable of handling. In this way the kernel handles the load balancing for us.

Re: GitHub was down

#247

Post by dang »

bitbasher wrote: post #130
red_Seashell_32 wrote: post #108

Earlier quoted context omitted.

Wait. You use github pages for something or actually work on it?

I use it for something. I had a github page that was public, but it was made private and the DNS config was removed. Fast forward to today. I made the private repo public again and forced a deploy of the page without making a new commit. It said the DNS config was incomplete, so I tweaked it and hit "check again" and github went down. Probably unrelated, but the timing was spooky.

Sorry for the offtopicness - would you mind emailing hn@ycombinator.com so I can check in with you about a couple things regarding https://news.ycombinator.com/item?id=41221186?

Re: GitHub was down

#248

Post by intelVISA »

ergocoder wrote: post #176
fragmede wrote: post #118

Earlier quoted context omitted.

from where? they don't only have one load balancer, so you'd still have the problem of the page showing green when it's not loading for some folk?

At Github's scale, why wouldn't they put a ping monitor from every continent at least? Then, you would show the status based on the continent.

That would be self-defeating given that it's a Rails app.

Re: GitHub was down

#250

Post by fragmede »

ergocoder wrote: post #176
fragmede wrote: post #118

Earlier quoted context omitted.

from where? they don't only have one load balancer, so you'd still have the problem of the page showing green when it's not loading for some folk?

At Github's scale, why wouldn't they put a ping monitor from every continent at least? Then, you would show the status based on the continent.

Where on the continent? GitHub is undoubtedly doing blackbox testing internally and has multiple such monitors but that's not going to capture every customer's route to them, leading to the same problem - customers experience GitHub being down, despite monitoring saying it's mostly up. Thus the impass. Even doing whitebox testing, where you know the internals and can this place sensors intelligently, even just for ingress, you're still at the mercy of the Internet.

If a sensor that's basically in the same datacenter says you're up, but the route into the datacenter is down, then what? multiply this by the complexity of the whole site, and monitoring it all with 100% fidelity is impossible. Not that it's not worth it to try, there's a team at GitHub that works on monitoring, but beyond motivation about keeping the SLA up, as a customer, unless you notice it's down, is it really down? In a globally distributed system, downtime, except for catastrophic downtime like this, is hard to define on a whole-site basis for all customers.

Post reply on HN