Don’t host yourself, it’s impossible to meet the reliability of the professionals
... HN is mostly professional software developers though. Running a GitLab instance really isn't hard.
GitHub was down again
41–50 of 125 posts
Re: GitHub was down again
#42I was seeing PRs failing to update & webhooks failing to trigger upon pushing code for 30 minutes before GH's status page acknowledged anything. I'm surprised they don't have monitoring in place that would catch webhooks failing within minutes of the failure beginning.
Had the same issue this morning. The lagging status always causes the issue of "is it you, me or GitHub?" snaffoos. Really annoying to have these issues so consistently. Would switch to gitea or similar in a moment given the choice.
Re: GitHub was down again
#43I was seeing PRs failing to update & webhooks failing to trigger upon pushing code for 30 minutes before GH's status page acknowledged anything. I'm surprised they don't have monitoring in place that would catch webhooks failing within minutes of the failure beginning.
Had the same issue this morning. The lagging status always causes the issue of "is it you, me or GitHub?" snaffoos. Really annoying to have these issues so consistently. Would switch to gitea or similar in a moment given the choice.
Just a friendly correction: `SNAFU:' Situation Normal: All Fucked Up
Re: GitHub was down again
#44How is GitLab like in terms of downtime? I looked at their status history page and I'm seeing a lot of incidents but it's hard to figure out what it actually means.
Thats why the self hosted options are there -- and why GitLab has a competitive advantage in this sense. Cloud solutions are great -- however they have a golden rule, don't go down ever. This is seriously damaging to GitHubs reputation.
> Thats why the self hosted options are there -- and why GitLab has a competitive advantage in this sense.
GitHub has GitHub Enterprise
> Cloud solutions are great -- however they have a golden rule, don't go down ever.
Oh where oh where can I sign up for this mythical unicorn cloud service?
Re: GitHub was down again
#45Don’t host yourself, it’s impossible to meet the reliability of the professionals
I used to agree. Now I work with a locally hosted github. It is down all the time and sometimes it just deletes all of the work from the past day. I thought it wasn't possible to do much worse, but I was obviously wrong.
Re: GitHub was down again
#46Where does github publish post-moderms of downtime? I only see things like "We have deployed a fix and are monitoring recovery." in the github status history which doesn't provide details.
These posts contain some details about incidents over the last few months:
https://github.blog/2020-07-08-introducing-the-github-availa...
https://github.blog/2020-05-22-april-service-disruptions-ana...
Re: GitHub was down again
#47I was seeing PRs failing to update & webhooks failing to trigger upon pushing code for 30 minutes before GH's status page acknowledged anything. I'm surprised they don't have monitoring in place that would catch webhooks failing within minutes of the failure beginning.
At large bureaucratic organisations there's often political implications of changing the official status, so often it lags behind reality until it cannot be swept under the rug anymore Not saying it's right, just an observation
What's the point in having a status page if it's a political artifact? In that case, it serves zero customer value.
Re: GitHub was down again
#48Earlier quoted context omitted.
That's the most annoying thing. Usually when I get notifications from monitoring about some issue, the first thing I do is check the vendor or provider's status page to see whether it's an issue on their end. If there's nothing, I go and investigate. Recently, more and more of them take 10-15 minutes until they mention a service outage. I don't work in super HA, I don't want to get an alarm because a single ping fail…
Idea for an startup: Paying a service to do independent health checks to popular services with the ability to select the services i would like to be notified of their health status.
Pagerduty,OpsGenie, etc?
Re: GitHub was down again
#49Earlier quoted context omitted.
As a CTO responsible for an often failing eCommerce website that lost millions when down and which I took over, I fell in the same trap of trying to sweep things under the rug. Until I decided to no longer do that and my life improved considerably.
Yeah it's very frustrating - especially if your customers are technical, they are seeing the errors and the status page says everything is fine. I've seen status pages and error counts tied to bonuses, which only caused a giant mess of bad incentive alignment and internal lies, customers are unhappy, developers are unhappy, management are lying to upper management, it's so much easier to focus efforts on real problem…
Do these companies not have live error reporting and tracing? Like surely github got alerts that things weren't working? Why don't they just hookup their status page and their alerts? Or is it a political/relationship thing, and they want to have a human give out the status page updates?
This could have been caught with a cron job and some curl requests :\
Re: GitHub was down again
#50GitHub actions downtime is becoming painful for us. Having been lured on there with 10,000 included minutes which they shortly thereafter dropped to 3,000 I feel aggrieved paying for overages incurred from actions regularly shitting the bed.
Github actions has been a huge let down for me. Between uptime issues and the lack of support for so many basic CI features is killing it for me (and has been for a year).
The only reason we're using it is because it's free..