GitHub was down again • Hacker News Board

Re: GitHub was down again

#11

Post by luckylion » Wed, Jul 15, 2020, 4:46 PM UTC

rollulus wrote: post #8

uberman wrote: post #6

Earlier quoted context omitted.

The status page clearly states (when I look) that: Incident on 2020-07-15 15:41 UTC We are investigating reports of degraded performance. Posted 9 minutes ago. Jul 15, 2020 - 15:41 UTC

Currently it does, indeed. From my Slack logs, at 15:00 UTC I noticed problems. I'm pretty sure that message is manually created, at least 41 minutes after the fact.

That's the most annoying thing. Usually when I get notifications from monitoring about some issue, the first thing I do is check the vendor or provider's status page to see whether it's an issue on their end. If there's nothing, I go and investigate.

Recently, more and more of them take 10-15 minutes until they mention a service outage. I don't work in super HA, I don't want to get an alarm because a single ping failed etc, so I'm lenient and have a few minutes of a delay in alarms. If I'm writing an internal incident report before the official status page is updated, that's bad.

This seems similar: external users noticing the outage and posting on HN before GitHub notices & acknowledges it.

Re: GitHub was down again

#12

Post by Axsuul » Wed, Jul 15, 2020, 4:46 PM UTC

originof wrote: post #2

The Latest commits don't appear in the commits tab

Same

Re: GitHub was down again

#13

Post by arilotter » Wed, Jul 15, 2020, 4:47 PM UTC

I was seeing PRs failing to update & webhooks failing to trigger upon pushing code for 30 minutes before GH's status page acknowledged anything. I'm surprised they don't have monitoring in place that would catch webhooks failing within minutes of the failure beginning.

Re: GitHub was down again

#14

Post by drcongo » Wed, Jul 15, 2020, 4:51 PM UTC

They're probably using the Facebook SDK /s

Re: GitHub was down again

#15

Post by zelly » Wed, Jul 15, 2020, 4:51 PM UTC

https://gitea.io/en-us/

https://git.zx2c4.com/cgit/

https://about.gitlab.com/install/?version=ce

Re: GitHub was down again

#16

Post by cameronfraser » Wed, Jul 15, 2020, 4:59 PM UTC

Why is outage history pre-acquisition removed from their history? If you try to go back in time it seems they only retain history up to a couple months after the acquisition. Is this just a 2 year retention policy or something being swept under the rug?

Re: GitHub was down again

#17

Post by malux85 » Wed, Jul 15, 2020, 5:01 PM UTC

arilotter wrote: post #13

I was seeing PRs failing to update & webhooks failing to trigger upon pushing code for 30 minutes before GH's status page acknowledged anything. I'm surprised they don't have monitoring in place that would catch webhooks failing within minutes of the failure beginning.

At large bureaucratic organisations there's often political implications of changing the official status, so often it lags behind reality until it cannot be swept under the rug anymore

Not saying it's right, just an observation

Re: GitHub was down again

#18

Post by yizhang7210 » Wed, Jul 15, 2020, 5:01 PM UTC

Ugh the last month has been pretty difficult. Hope they get better soon.

Re: GitHub was down again

#19

Post by marcinzm » Wed, Jul 15, 2020, 5:09 PM UTC

How is GitLab like in terms of downtime? I looked at their status history page and I'm seeing a lot of incidents but it's hard to figure out what it actually means.

Re: GitHub was down again

#20

Post by donatj » Wed, Jul 15, 2020, 5:09 PM UTC

It's been having issues all day. Wanted to show a coworker some changes I was proposing but the site wouldn't show the changes I'd pushed to my pull request. Ended up just having him pull the changes.

FWIW the git backend always seems rock solid in comparison to the front end they have displaying it.