GitHub was down again • Hacker News Board

Re: GitHub was down again

#51

Post by bob1029 » Wed, Jul 15, 2020, 5:36 PM UTC

Here we are again. Me taking a break on Hackernews because all my webhooks and pull requests are fucked and I have no idea where my devops tools are relative to what the real state of affairs is. I have pretty much had enough of this. It is too disruptive to our process. It is causing fragility and loss of confidence in our build pipeline.

At this point, we would probably be better off just bolting some lightweight git solution onto our devops tools (which are 100% custom in-house developed), rather than fighting with some more-durably-hosted offering of GitHub, et. al.

Anyone who posts that "but you cant make it more reliable than microsoft" line is not thinking about the dependencies between systems and the considerable impact incurred on a service just by virtue of it being a publicly-accessible platform without any cost barrier to entry. Sure, bringing it in house might bring additional difficulties, but I think I can eliminate a shitload of existing difficulties if we moved from webhooks across the public internet to a direct method invocation within the same binary image.

Re: GitHub was down again

#52

Post by opportune » Wed, Jul 15, 2020, 5:38 PM UTC

JMTQp8lwXL wrote: post #47

malux85 wrote: post #17

Earlier quoted context omitted.

At large bureaucratic organisations there's often political implications of changing the official status, so often it lags behind reality until it cannot be swept under the rug anymore Not saying it's right, just an observation

What's the point in having a status page if it's a political artifact? In that case, it serves zero customer value.

It’s still an indicator that “no you’re not crazy, we’re having issues on our end” but not a foolproof one. It’s kind of sad that twitter is usually the best place to confirm an outage as it begins, rather than the software providers themselves. I assume if they actually exposed global availability metrics in most cases it would not look as good as they would want it to

Re: GitHub was down again

#53

Post by dabeeeenster » Wed, Jul 15, 2020, 5:38 PM UTC

bob1029 wrote: post #51

Here we are again. Me taking a break on Hackernews because all my webhooks and pull requests are fucked and I have no idea where my devops tools are relative to what the real state of affairs is. I have pretty much had enough of this. It is too disruptive to our process. It is causing fragility and loss of confidence in our build pipeline. At this point, we would probably be better off just bolting some lightweight g…

We've been self hosting Gitlab CE for a couple of years. Its been great. No downtime, upgrades are seamless, fast, works.

Re: GitHub was down again

#54

Post by mattstrayer » Wed, Jul 15, 2020, 5:39 PM UTC

bob1029 wrote: post #51

Here we are again. Me taking a break on Hackernews because all my webhooks and pull requests are fucked and I have no idea where my devops tools are relative to what the real state of affairs is. I have pretty much had enough of this. It is too disruptive to our process. It is causing fragility and loss of confidence in our build pipeline. At this point, we would probably be better off just bolting some lightweight g…

or, you know, host a gitlab instance yourself & call it a day

Re: GitHub was down again

#55

Post by michaelwda » Wed, Jul 15, 2020, 5:41 PM UTC

atraac wrote: post #33

carstenhag wrote: post #10

Earlier quoted context omitted.

Also having outages at Azure DevOps Pipelines every other month or so it seems. And that's paying - for hours there's no mention on the status page and we are stuck there, not being able to merge PRs or release our app in the standard way.

This is weird because we haven't encountered any real issues with agents in Azure DevOps pipelines. I think we maybe had a single downtime in last 6 months. They recently removed .NET Core 2.2 SDK without any notice and broke our builds but that's another thing.

I couldn't find ANYTHING about the 2.2 removal. Just one day our unit test projects stopped working. I understand that 2.1 is LTS, but still.

Re: GitHub was down again

#56

Post by hundchenkatze » Wed, Jul 15, 2020, 5:43 PM UTC

zelly wrote: post #15

https://gitea.io/en-us/ https://git.zx2c4.com/cgit/ https://about.gitlab.com/install/?version=ce

I've been having a great experience with https://sourcehut.org/ as well.

Re: GitHub was down again

#57

Post by stunt » Wed, Jul 15, 2020, 5:45 PM UTC

At the current rate, Whatever you host will have a better uptime than Github.

Re: GitHub was down again

#58

Post by niftylettuce » Wed, Jul 15, 2020, 5:46 PM UTC

There have been at least three major outages, e.g. git clone of a repo, in the past week alone. All three of which have been unreported (and NOT shown on their incident page), but I have email confirmation from GitHub support of these issues. It's almost time to switch to Gitlab. I have hundreds of repositories, organizations, and packages to transfer, while it will be daunting... I need reliability. I have several paid GitHub orgs and accounts as well.

Re: GitHub was down again

#59

Post by rydre » Wed, Jul 15, 2020, 5:48 PM UTC

I really want to move to GitLab but its UI is atrocious... like too mobile phone looking on desktop

Re: GitHub was down again

#60

Post by mardifoufs » Wed, Jul 15, 2020, 5:50 PM UTC

ketzu wrote: post #29

cameronfraser wrote: post #16

Why is outage history pre-acquisition removed from their history? If you try to go back in time it seems they only retain history up to a couple months after the acquisition. Is this just a 2 year retention policy or something being swept under the rug?

I can go back all the way to 2010: https://imgur.com/DsSKcFV

Wow, it used to be so much more detailed! I get they probably can't have that level of "casual" disclosure now that they are so big, but man the current status updates just feel so... useless and unhelpful in comparison.