Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

GitHub was down

githubstatus.com

Post reply on HN
Print view View original

111–120 of 171 posts

Re: GitHub was down

#111

Post by thanatropism »

allanbreyes wrote: post #97
cle wrote: post #68

Earlier quoted context omitted.

> How do we calculate Uptime? > Our Uptime calculation is based on the percentage of successful requests we serve through our web, API, and Git client interfaces. Just curious, how do they measure this? What is the actual calculation?

> What is the actual calculation? Not answering this directly, but the paper Meaningful Availability [0] released recently really changed my opinion on how to calculate and visualize availability. There's a discussion on HN as well [1]. [0]: https://www.usenix.org/system/files/nsdi20spring_hauer_prepu... [1]: https://news.ycombinator.com/item?id=22424173

That was insightful to some point. But, of course, the relevant metric is "expected availability" -- before I decide to go on the service; therefore, not the same as "customers served". If I have to think about downtime then my experience is degraded; moreso if I have to delay and batch planned interactions (all of which will later count as successful!)

[Edit: To the point: a high rate of randomly-timed failures is a kind of degraded experience, but not as critical as blocky patches of downtime. A 1% rate of randomly-timed failures is much much much preferred than having the service go out three straight days every February.]

Also: uptime is not the same as "customer delight". It's all about time.

Re: GitHub was down

#112

Post by jws »

cle wrote: post #68
waynenilsen wrote: post #51

Don't forget to check your SLAs Enterprise = 99.95% (quarterly) https://help.github.com/en/github/site-policy/github-enterpr... They're having a bad February but January was good. We will see what March has in store

> How do we calculate Uptime? > Our Uptime calculation is based on the percentage of successful requests we serve through our web, API, and Git client interfaces. Just curious, how do they measure this? What is the actual calculation?

Our Uptime calculation is based on the percentage of successful requests we serve through our web, API, and Git client interfaces.

So when a customer finds a broken service it is in their financial best interest to repeatedly hammer the broken service and drive down the uptime calculation to trigger their rebate.

Just an observation, not a suggestion. I’d fire any customer I found doing this.

Re: GitHub was down

#113

Post by mattigames »

iso947 wrote: post #87
pdelgallego wrote: post #13

Earlier quoted context omitted.

"You can't legislate against failure, but you can focus on fast detection and response" -- Chris Pinkham

It’s amazing how this is accepted in the software world. Move fast and break things, such a different philosophy to other areas.

I mean, that's rather disputable: The Apollo 1 exploded, medical mistakes have a toll of 250.000 deaths per year in the US alone; among many other serious mistakes on vast different areas, I think unreliability is unfortunately a constant on the human race.

Re: GitHub was down

#114

Post by chrisweekly »

Cthulhu_ wrote: post #84
amelius wrote: post #60

Earlier quoted context omitted.

The interesting thing is that Git is entirely non-centralized, so in theory they could simply redirect to servers onto which the data has been mirrored.

Git is, but the APIs and all the services they provide around it aren't. That said, I think it's a bit weird that they don't store the data of the services around the code itself in git, like they do with e.g. sites. That way you'd have an `issues` branch that you could still access if github is down. But that would probably pave the way for easy migrations away from Github.

>"But that would probably pave the way for easy migrations away from Github."

bingo

Re: GitHub was down

#115

Post by _salmon »

pdelgallego wrote: post #13
savrajsingh wrote: post #8

No matter how many talented engineers you have on staff, your entire service can still go down. Let's pause and reflect on that. ;)

"You can't legislate against failure, but you can focus on fast detection and response" -- Chris Pinkham

"I will not be harassed in my own private domicile"

-- Jesse Pinkman

Re: GitHub was down

#116

Post by natfriedman »

Sincere apologies to all GitHub users for the downtime this morning, and the brief outages last week as well. We take reliability very seriously, and will publish a full RCA in the near future.

Re: GitHub was down

#117

Post by allanbreyes »

thanatropism wrote: post #111
allanbreyes wrote: post #97

Earlier quoted context omitted.

> What is the actual calculation? Not answering this directly, but the paper Meaningful Availability [0] released recently really changed my opinion on how to calculate and visualize availability. There's a discussion on HN as well [1]. [0]: https://www.usenix.org/system/files/nsdi20spring_hauer_prepu... [1]: https://news.ycombinator.com/item?id=22424173

That was insightful to some point. But, of course, the relevant metric is "expected availability" -- before I decide to go on the service; therefore, not the same as "customers served". If I have to think about downtime then my experience is degraded; moreso if I have to delay and batch planned interactions (all of which will later count as successful!) [Edit: To the point: a high rate of randomly-timed failures is a…

> a high rate of randomly-timed failures is a kind of degraded experience, but not as critical as blocky patches of downtime

Do you think that's an accurate generalization for all software and business contexts? I think a novel insight about the paper is that windowed user uptime is able to visualize the differences. (See Figure 20 from the paper.)

Re: GitHub was down

#118

Post by latchkey »

natfriedman wrote: post #116

Sincere apologies to all GitHub users for the downtime this morning, and the brief outages last week as well. We take reliability very seriously, and will publish a full RCA in the near future.

Thanks Nat! Keep up the good work and thanks for contributing here. Github is still my favorite. =)

Re: GitHub was down

#119

Post by johannes1234321 »

allanbreyes wrote: post #117
thanatropism wrote: post #111

Earlier quoted context omitted.

That was insightful to some point. But, of course, the relevant metric is "expected availability" -- before I decide to go on the service; therefore, not the same as "customers served". If I have to think about downtime then my experience is degraded; moreso if I have to delay and batch planned interactions (all of which will later count as successful!) [Edit: To the point: a high rate of randomly-timed failures is a…

> a high rate of randomly-timed failures is a kind of degraded experience, but not as critical as blocky patches of downtime Do you think that's an accurate generalization for all software and business contexts? I think a novel insight about the paper is that windowed user uptime is able to visualize the differences. (See Figure 20 from the paper.)

It certainly is not - a trade might not care about 99% of the time, but the exact moment they want to do a trade the system must work.

Whereas of some GitHub request fails and I retry it's a minor annoyance, but in most cases I won't even know whether that was GitHub's flaw, my local system or some networking in between.

Re: GitHub was down

#120

Post by MuffinFlavored »

natfriedman wrote: post #116

Sincere apologies to all GitHub users for the downtime this morning, and the brief outages last week as well. We take reliability very seriously, and will publish a full RCA in the near future.

What did you guys deploy/what scale tipping point did you guys hit that caused the past 3 days of problems?

At my job, if something we go wrong... management just tells us roll it back. That always fixes the problem, right? :P

Post reply on HN