Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

GitHub Outage

news.ycombinator.com

Post reply on HN
Print view View original

71–80 of 87 posts

Re: GitHub Outage

#71

Post by judge2020 »

naikrovek wrote: post #29
adamscybot wrote: post #11

Earlier quoted context omitted.

In my experience its often the case the status site does not reflect reality -- until someone intervenes.

that's the whole point of a manually updated status page. you don't want automation to update it because that automation can fail. automation likely caused the outage you want to know more about. you also don't want your automation guessing at what the problem is, or what the effects are. you want real info from a real person even if it isn't given to you the millisecond you look for it. this is why status pages aren…

Yet Discord has an “API Response Time” graph[0] and Reddit has a “5xx error rate” graph[1]. No, it doesn’t automatically create incidents, but it’s nice to confirm an issue is happening site-wide after experiencing it.

Actually looks like the metrics part of Reddit’s status page broke over 2 weeks ago

0: https://discordstatus.com/

1: https://www.redditstatus.com/

Re: GitHub Outage

#72

Post by capableweb »

rozenmd wrote: post #63
omn1 wrote: post #2

Even though the status page ( https://www.githubstatus.com/ ) shows no issues, I'm still getting the occasional 500. It seems to be happening quite irregularly. They are possibly facing a lot of load.

I monitor GitHub externally here: https://github.onlineornot.com/ Seems like a huge spike in load.

Load as in load average? Or load as in traffic?

Spikes in request latency can be because of bunch of stuff, including more traffic, but in my experience, it's usually around non-existing optimizations for some data structure that got triggered after N items or new deploys containing code that wasn't as optimal as the author of the code thought. Especially when dealing with distributed systems, where sub-optimal code in one part can cascade performance issues to various parts in the system.

Re: GitHub Outage

#73

Post by capableweb »

blueflow wrote: post #22

Do we need to create a HN post for every outage? It happens every other week.

In the beginning of status pages, most of them were automatic one way or another, or engineers quickly threw up "We know of the problem, stay tuned" messages there.

But soon after, legal/executive team got ownership of them apparently, and the status pages are no longer automatically showing downtime/response time and notice about when things are actually down can take a while.

So I think it's nice that there is at least one place where I can see if it's a problem on my end, or if it's global. It helps to remove some frustration at least.

Re: GitHub Outage

#74

Post by capableweb »

chrisseaton wrote: post #62
capableweb wrote: post #60

Earlier quoted context omitted.

Self-hosting Git is easy, throw up a ssh server and point git to it. Self-hosting everything else GitHub does is harder. Which is why they are building out all of those things, they don't want people to move to other places so easily. Hopefully these constant outages makes more developers pissed off that issues are not stored in git as well, and start working on tooling to solve this shitty problem once and for all.…

> Self-hosting everything else GitHub does is harder. You can self-host the whole of GitHub can’t you?

As far as I know, you can self-host an enterprise version of GitHub, but it'll still be available from one location (the server you deploy it on). I cannot run it locally and federate with my colleagues instances for example, just like I can with Git.

Re: GitHub Outage

#75

Post by bilalq »

How reliable are Github cron action workflows? I set one up to run every 15 minutes recently, but it seems to actually be running closer to once an hour.

Re: GitHub Outage

#76

Post by bowsamic »

MattIPv4 wrote: post #33
bowsamic wrote: post #14

Earlier quoted context omitted.

Status pages are absolutely useless. I've never seen them accurately reflect an outage

You are missing the point of a status page. They're not automatic things that tell you instantly when something is down -- that'd be pretty pointless, you already know it's down. They're updated by the folks responding to the incident, so you know they know there's an issue and that they're looking into it.

Then it's not an operational status, it's an engineering status. Clearly it is very misleading. I think most people, even devs, think these pages are supposed to reflect the current situation. Btw the Github one still doesn't

Re: GitHub Outage

#77

Post by grepfru_it »

judge2020 wrote: post #71
naikrovek wrote: post #29

Earlier quoted context omitted.

that's the whole point of a manually updated status page. you don't want automation to update it because that automation can fail. automation likely caused the outage you want to know more about. you also don't want your automation guessing at what the problem is, or what the effects are. you want real info from a real person even if it isn't given to you the millisecond you look for it. this is why status pages aren…

Yet Discord has an “API Response Time” graph[0] and Reddit has a “5xx error rate” graph[1]. No, it doesn’t automatically create incidents, but it’s nice to confirm an issue is happening site-wide after experiencing it. Actually looks like the metrics part of Reddit’s status page broke over 2 weeks ago 0: https://discordstatus.com/ 1: https://www.redditstatus.com/

GitHub has such a graph too, but limited to internal employees only. In fact that’s how they detect failures with their deployments, elevated 500 statuses. Some individual teams will have their own dashboard, but some do not (like k8s upgrades) and they only monitor 500s.

Rest assured someone is looking into this problem right now

Re: GitHub Outage

#78

Post by corford »

Yep, noticed it with comments on an issue (had timeouts while submitting but it eventually went through).

Now 30 mins later, i've refreshed the issue and see that my reply and the comment I was replying too (by another user) are both gone. Hopefully, it's eventually consistent and these comments will re-appear later.

Re: GitHub Outage

#80

Post by naikrovek »

judge2020 wrote: post #71
naikrovek wrote: post #29

Earlier quoted context omitted.

that's the whole point of a manually updated status page. you don't want automation to update it because that automation can fail. automation likely caused the outage you want to know more about. you also don't want your automation guessing at what the problem is, or what the effects are. you want real info from a real person even if it isn't given to you the millisecond you look for it. this is why status pages aren…

Yet Discord has an “API Response Time” graph[0] and Reddit has a “5xx error rate” graph[1]. No, it doesn’t automatically create incidents, but it’s nice to confirm an issue is happening site-wide after experiencing it. Actually looks like the metrics part of Reddit’s status page broke over 2 weeks ago 0: https://discordstatus.com/ 1: https://www.redditstatus.com/

> Yet Discord has an “API Response Time” graph[0] and Reddit has a “5xx error rate” graph[1].

that's awesome. doesn't exist for github, yet. would be nice if it does come.

Post reply on HN