Live data from Hacker News

Tell HN: GitHub is down (Update: Back online now)

news.ycombinator.com

61–70 of 73 posts

Re: Tell HN: GitHub is down (Update: Back online now)

#61
post #5

According to GH Status ( https://www.githubstatus.com/ ), everything is fine. Gotta love functional status pages. edit: Nevermind, they just reported "degraded performance" for GitHub Actions, Issues, and Pull Requests.

Quoted post unavailable.

1) SLAs are almost always customer initiated (see "Customer Must Request Financial Credit." in https://cloud.google.com/appengine/sla for example)

2) The SLA isn't tied to the status page at all, a customer can request an SLA refund for any reason provided they have proof of damages.

TL;DR: status pages are a helpful tool to let folks know something's wrong and that the team isn't asleep at the wheel, not a legally binding contract.

Re: Tell HN: GitHub is down (Update: Back online now)

#62
post #41
post #16

Yup. I've been disrupted by GitHub today, and DockerHub the other day. Crude reminder that the cloud is some company's computer.

My own computer wouldn’t be much better.

Same. My desktop won't even currently power on after a move... Looks like I get to reseat all of my RAM tonight :)

Re: Tell HN: GitHub is down (Update: Back online now)

#63
post #60

Earlier quoted context omitted.

Quoted post unavailable.

I can't imagine the status page is mentioned anywhere in the contracts. I would think that the only language that is included is that it's the customers responsibility to track and notify the vendor (Github) about any downtime, and to do that within 48 hours of the downtime having occured. And only then will the customer be eligible for compensation, probably in the form of free service credits. My personal take is t…

The SLA in their generic terms states that the calculation is based on their monitoring: https://github.com/customer-terms

A process that involved arguing about what a status page said and when doesn't make a lot of sense if they have logs of what happened.

Re: Tell HN: GitHub is down (Update: Back online now)

#64
post #31

Incident is up now - https://www.githubstatus.com/incidents/fz1bdbw24y81 > We are investigating reports of degraded performance for GitHub Actions, Issues, and Pull Requests.

Degraded Performance? Their freaky homepage is borked: https://github.com/ yields a 500 error haha.

100% degraded is degraded.

Re: Tell HN: GitHub is down (Update: Back online now)

#65
post #5

According to GH Status ( https://www.githubstatus.com/ ), everything is fine. Gotta love functional status pages. edit: Nevermind, they just reported "degraded performance" for GitHub Actions, Issues, and Pull Requests.

i wonder how they even work? Like, do they just display a green css button instead of _actually_ doing a healthcheck?

Yes they do, because very rarely would such a healthcheck kind of setup actually work in practice, at a large enough size, for a user-facing dashboard. If you want a healthcheck, look at a Grafana dashboard, not a status page.

By the way, I don't know of a single place where this isn't the case, where a human signs off on and updates the status page during large events (at least at the final decision.) Some of it will be automated, sure, like red flags being raised to operators. But at a certain point it is not possible to automate this in some level to achieve second-level accuracy or whatever; the system is rarely (if ever) in a binary state of "working perfectly" or "not working", but somewhere in between. You can't just fire off a big red error bar every time a blip occurs at a place like GitHub. The system is constantly "in motion". The logical conclusion is to just expose your 50+ Grafana dashboards publicly to every user. Isn't that the most honest "overview" of what is happening with your product? Except this often can't tell them useful things either.

People on here will also mumble about SLAs but if a customer wants a kickback or is seriously worried about events like these, they're generally talking to account managers, not posting on internet forums. That said, a lot of them get weaselly about that stuff unless you're already negotiating prices with an AM in the first place...

Re: Tell HN: GitHub is down (Update: Back online now)

#66

Earlier quoted context omitted.

i wonder how they even work? Like, do they just display a green css button instead of _actually_ doing a healthcheck?

Probably requires manual updates. It seems like more and more places have moved to this paradigm now that status pages are tied to SLAs which are tied to money. One might call it the politicization of status pages.

Politicization, yes; I've never heard of SLAs being tied to status pages. It is like pulling teeth to get most cloud providers to credit the account when they don't meet SLA, and one always has to ask for it; heaven forbid if credits were paid out automatically when service wasn't rendered.

Or you get weasel-worded out of it. I had a cloud provider deny a service credit; the SLA stated that the service was only out of SLA if it didn't return 2xx. Well, the API returned "2xx Accepted — your request is being processed", and you could use the API to query the job, and the job … never finished or made any progress at all. But the API returned 2xx the entire time, so that was "within SLA".

Re: Tell HN: GitHub is down (Update: Back online now)

#68
post #5

According to GH Status ( https://www.githubstatus.com/ ), everything is fine. Gotta love functional status pages. edit: Nevermind, they just reported "degraded performance" for GitHub Actions, Issues, and Pull Requests.

Didn't they have actual error rate graphs on that page back in a day?

Yes, and some response times too. It was actually useful.

Re: Tell HN: GitHub is down (Update: Back online now)

#69
post #58

Earlier quoted context omitted.

Why manually update vs automatically?

Not many places I have worked for allowed for this to be automatic. A lot of it was so they could provide a coherent explanation as to what the current state of internal attention was directed at vs what everyone can plainly see.

I guess this makes sense, but I don't understand why you couldn't have it change status automatically, and still allow a person to go in after and manually add an explanation.

Re: Tell HN: GitHub is down (Update: Back online now)

#70
post #58

Earlier quoted context omitted.

Not many places I have worked for allowed for this to be automatic. A lot of it was so they could provide a coherent explanation as to what the current state of internal attention was directed at vs what everyone can plainly see.

I guess this makes sense, but I don't understand why you couldn't have it change status automatically, and still allow a person to go in after and manually add an explanation.

Mostly because it's more work to manually remove a bunch of spurious robotically added statuses (which look bad, because you're either down more than customers are noticing or your detection is flawed) than it is to manually add and remove real ones.
Post reply on HN