Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

GitHub was down

githubstatus.com

Post reply on HN
Print view View original

81–90 of 171 posts

Re: GitHub was down

#81

Post by School-Cotton »

ithkuil wrote: post #34
chimprich wrote: post #4

Earlier quoted context omitted.

If Github is down, thousands of programmers suddenly have nothing better to do.

I rolled my eyes twice, then merged a few PRs manually and moved on with my day. (i.e. the git server itself and all the API required to interact with the CI automation _appears_ to work just fine)

The actual server was broken for me:

  $ git push
  Enumerating objects: 26, done.
  Counting objects: 100% (26/26), done.
  Delta compression using up to 8 threads
  Compressing objects: 100% (15/15), done.
  Writing objects: 100% (15/15), 1.49 KiB | 1.49 MiB/s, done.
  Total 15 (delta 12), reused 0 (delta 0)
  remote: Resolving deltas: 100% (12/12), completed with 10     local objects.
  remote: Internal Server Error
  To git+ssh://github.com//
   ! [remote failure]    wip -> wip (remote failed to report status)
  error: failed to push some refs to 'git+ssh://git@github.com//'

Re: GitHub was down

#83

Post by tribaal »

amelius wrote: post #60
savrajsingh wrote: post #8

No matter how many talented engineers you have on staff, your entire service can still go down. Let's pause and reflect on that. ;)

The interesting thing is that Git is entirely non-centralized, so in theory they could simply redirect to servers onto which the data has been mirrored.

Yes, git is.

The issues, comments, PRs, wikis etc... that we all came to depend on aren't.

Re: GitHub was down

#84

Post by Cthulhu_ »

amelius wrote: post #60
savrajsingh wrote: post #8

No matter how many talented engineers you have on staff, your entire service can still go down. Let's pause and reflect on that. ;)

The interesting thing is that Git is entirely non-centralized, so in theory they could simply redirect to servers onto which the data has been mirrored.

Git is, but the APIs and all the services they provide around it aren't.

That said, I think it's a bit weird that they don't store the data of the services around the code itself in git, like they do with e.g. sites. That way you'd have an `issues` branch that you could still access if github is down.

But that would probably pave the way for easy migrations away from Github.

Re: GitHub was down

#85

Post by bradstewart »

Benjammer wrote: post #74
cle wrote: post #68

Earlier quoted context omitted.

> How do we calculate Uptime? > Our Uptime calculation is based on the percentage of successful requests we serve through our web, API, and Git client interfaces. Just curious, how do they measure this? What is the actual calculation?

What do you mean? All modern enterprise analytics/monitoring solutions are going to be able to give you some kind of top-level "request success rate" metric. I assume they mostly just lean into whatever monitoring tooling they have set up. What kind of "calculation" are you imagining here? Like a very specific SRE formula for availability windows or something?

Depending on what part of the system is down, how do you know you even got a request to mark as failed?

Re: GitHub was down

#86

Post by Cthulhu_ »

victords wrote: post #62
brainwipe wrote: post #52

Earlier quoted context omitted.

Correct. Availability is 99.99. https://aws.amazon.com/s3/storage-classes/

Regardless, the SLA that AWS (& other cloud providers) meet is quite impressive.

But what do they offer if they don't meet it? I mean a discount on your monthly bill somehow doesn't sound like it would cover a potential loss.

Re: GitHub was down

#87

Post by iso947 »

pdelgallego wrote: post #13
savrajsingh wrote: post #8

No matter how many talented engineers you have on staff, your entire service can still go down. Let's pause and reflect on that. ;)

"You can't legislate against failure, but you can focus on fast detection and response" -- Chris Pinkham

It’s amazing how this is accepted in the software world. Move fast and break things, such a different philosophy to other areas.

Re: GitHub was down

#89

Post by siffland »

amelius wrote: post #60
savrajsingh wrote: post #8

No matter how many talented engineers you have on staff, your entire service can still go down. Let's pause and reflect on that. ;)

The interesting thing is that Git is entirely non-centralized, so in theory they could simply redirect to servers onto which the data has been mirrored.

Every project i have been on where we built for availability and resilience has inevitably had at least one single point of failure. Usually it is something deemed non critical, but somehow can still bring the infrastructure down (A single DNS server at one of our production sites did this, we have 2 more accessible via a VPN tunnel, it was deemed if the production DNS went down the other two were still reachable, to bad the day it happened the tunnel was down too).

Also you have to deal with sysadmin error, i know us sysadmins are practically perfect in every way, but occasionally we make mistakes....big mistakes. ;)

So redirecting might not always be possible.....

Post reply on HN