Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

GitHub incident 2022-03-23

githubstatus.com

Post reply on HN
Print view View original

111–120 of 206 posts

Re: GitHub incident 2022-03-23

#111

Post by KronisLV »

xondono wrote: post #44
gjulianm wrote: post #32

Earlier quoted context omitted.

At my organization it's always been true. Setting up GitLab is fairly easy, in my company we do it and it's cheap (on-prem hosting is basically zero, and we had the IPs/domains already) and it hasn't given us too many headaches. I think last time I had to do something was maybe a few months ago when I restarted it so that it picked up the updated SSL certificate.

I’d say it depends, I run my own on prem server and gitlab was a PITA. Too many moving parts, updating took too much of my time, and I never felt “safe”. Moving to gitea solved all of those issues for me (thus far), now I’m looking into adding other stuff like CI through Drone.

Curiously, this was also my own experience!

I actually wrote a bit about the migration process, as well as the reasons for migrating over to Gitea, Nexus and Drone CI as opposed to using GitLab, GitLab Registry and GitLab CI: https://blog.kronis.dev/articles/goodbye-gitlab-hello-gitea-...

With containers, it's actually a pretty good experience that's not too hard to setup or manage.

Re: GitHub incident 2022-03-23

#113

Post by kleebeesh »

cortesoft wrote: post #100
kleebeesh wrote: post #88

Earlier quoted context omitted.

...What? $10 x 1000 = $10k / month. $10k x 12 = $120k. That is a new grad software engineer salary in any US city. You'd pay more than that for a single dev with the devops and security experience to keep GHE running and patched for 1000 devs.

The person was replying to a comment saying they spend more on a SINGLE HOUR of a dev's time than the monthly GH bill, which is not true for an org of more than 20 people or so (depending on hourly rate).

Ah, totally misread it. Thanks.

Re: GitHub incident 2022-03-23

#114

Post by devwastaken »

nimbius wrote: post #16

https://www.githubstatus.com/history 21 incident outages in just 3 months. At this rate the benefits of running your own gitea or gitlab are starting to become competitive.

And whom pays for fixing it? Downtimes of self hosted systems using external software can be far longer. GitHub, unlike Amazon and friends, doesn't lie about their downtime. Every saas has hundreds of downtime instances across the board every month. Some are small enough you don't see them. Yet the services still work exceptionally well - and when they don't they get fixed in a quick manner. What takes them an hour would take most private orgs a day.

Re: GitHub incident 2022-03-23

#115

Post by KronisLV »

ironmagma wrote: post #86
nimbius wrote: post #16

https://www.githubstatus.com/history 21 incident outages in just 3 months. At this rate the benefits of running your own gitea or gitlab are starting to become competitive.

We run Gitea at my company. In fact, we forked it. It could reeeaaaalllly use a rewrite. If anyone is even mildly ambitious about creating a new alternative to Github/Gitea, it's a great time to do that.

Another self-hosted project in the space that i've seen was GitBucket, although it runs on the JVM (not necessarily a bad thing, just different from Go): https://gitbucket.github.io/

Re: GitHub incident 2022-03-23

#116

Post by judge2020 »

mirekrusin wrote: post #80
thaeli wrote: post #60

Earlier quoted context omitted.

Also, looking at this it seems like GitHub isn't doing the common SaaS thing of just lying on their status page. Many providers, both internal and external, would look a lot worse if they had honest status pages.

They are green for good 15 minutes from first moment i see problems, not the first time, it happens actually quite often. Maybe that's the time they need to confirm/cross check/write status update, don't know.

They probably allow regular SREs to trigger an incident on the status page on their own, when the likes of AWS and other bigger cloud providers are rumored to need approval from a VP[0] to update the status page.

0: https://news.ycombinator.com/item?id=29475756

Re: GitHub incident 2022-03-23

#117

Post by mdellavo »

sonicggg wrote: post #110
nimbius wrote: post #16

https://www.githubstatus.com/history 21 incident outages in just 3 months. At this rate the benefits of running your own gitea or gitlab are starting to become competitive.

Come on, don't be so dramatic. This is not a 911 call center, people will survive these minor outages.

sure but three days in a row?

Re: GitHub incident 2022-03-23

#118

Post by zeku »

kleebeesh wrote: post #88
capitol_ wrote: post #82

Earlier quoted context omitted.

I think we pay about $10 per developer per month for github, and with about 1000 developers I would love that hourly rate.

...What? $10 x 1000 = $10k / month. $10k x 12 = $120k. That is a new grad software engineer salary in any US city. You'd pay more than that for a single dev with the devops and security experience to keep GHE running and patched for 1000 devs.

Just a bone to pick... new grad engineers in my US city started around 60-70k in 2018 when my college cohort graduated. Southern US...

Re: GitHub incident 2022-03-23

#119

Post by rvz »

Someone wrote: post #97
rvz wrote: post #24

Earlier quoted context omitted.

Well, I think I have said that since 2020 [0] and it is self-evident that you are better off self-hosting your own Git repo. If you can host a website you can do it. If GNOME, ReactOS, Wireguard, Linux Kernel Project, Mozilla, etc can do it, so can you. Or even use it as a backup / failsafe just in case. But going 'all in' on GitHub just doesn't make any sense anymore. [0] https://hn.algolia.com/?dateRange=all&page=1…

But who can host a website? I would be wary of hosting something that isn’t a 100% static site, out of fear of the amount of attention maintenance would take. Also, quite a few of the non-profits behind the projects you mentioned have multi-million dollar budgets that they can use to administer their git instance, if needed. I don’t think “if they can do it, you can” is a strong argument for those.

I don't recall ReactOS, or the creators of wireguard having 'multi million dollar budgets'. How is it that even projects like RedoxOS [0] are able to self-host on a GitLab instance using a subdomain, without giant budgets in the millions?

You don't need a 'multi-million dollar budget' to self-host a git repo and may of these open-source projects have been doing so even before GitHub existed for years. Even if they did have such a budget, there isn't an excuse left to self-host and avoid going 'all in' on GitHub.

At the very least I would expect something like what ReactOS is doing by having a self-hosted backup just in case GitHub goes down or vice-versa. [1]

Looks like that is proving to be useful.

[0] https://gitlab.redox-os.org/redox-os

[1] https://github.com/reactos/reactos#code-mirrors

Re: GitHub incident 2022-03-23

#120

Post by ejb999 »

TheRealPomax wrote: post #108
ejb999 wrote: post #84

Earlier quoted context omitted.

thats not the kind of progression you like to see - that is, error rates increasing over time instead of decreasing.

Only if you believe those numbers mean anything. What are the errors for ? Github has been adding lots of features and subproducts over the years, becoming a bigger and bigger platform as a result. What you want is the error-per-component, which may very well have actually gone down, with error spikes coming from "when github adds a completely new feature and it goes through a slew of incidents in its first year". Th…

Every place I have ever worked reported incidents going down would be good, not up.
Post reply on HN