GitHub incident 2022-03-23 • Hacker News Board

Re: GitHub incident 2022-03-23

#101

Post by hhh » Wed, Mar 23, 2022, 4:45 PM UTC

temp8964 wrote: post #92

nightpool wrote: post #61

Earlier quoted context omitted.

Self-hosted GitLab got a good callout yesterday from Microsoft, it appears to be a favorite of LAPSUS$: https://www.microsoft.com/security/blog/2022/03/22/dev-0537-... Self-hosting always increases the operational burden of making sure your systems are secure. Maybe you have the engineering resources to spend on patching everything immediately and conducting in-house pen tests, but for most companies it's much, much…

Not necessarily. Self-hosted services are protected by company firewall / VPN. They can setup very restrictive network access. They don't have the same level of risks as public services like GitHub or GitLab.

Establishing an entry point via VPN is Lapsus$ primary first step.

Re: GitHub incident 2022-03-23

#102

Post by TheRealPomax » Wed, Mar 23, 2022, 4:45 PM UTC

nimbius wrote: post #16

https://www.githubstatus.com/history 21 incident outages in just 3 months. At this rate the benefits of running your own gitea or gitlab are starting to become competitive.

But how many of those actually affected you? For example, no amount of issues around codespaces or github packages would impact my professional use of github, so whether there are 21 or 5000 or those parts get permanently taken offline makes no difference in what I need out of the platform.

How many core incidents? The part that affects whether you can even push to and pull from a repo, and access issues and PRs? Because everything else is nice to have, but you can do work perfectly fine without them if they go down for a few hours.

Re: GitHub incident 2022-03-23

#103

Post by megous » Wed, Mar 23, 2022, 4:45 PM UTC

gwbas1c wrote: post #74

nimbius wrote: post #16

https://www.githubstatus.com/history 21 incident outages in just 3 months. At this rate the benefits of running your own gitea or gitlab are starting to become competitive.

> At this rate the benefits of running your own gitea or gitlab are starting to become competitive When you host things yourself, you still have downtime. And, having worked with Github for over a decade, the actual disruption to my work is from downtime is much less than if I had to host my own. That being said: I briefly worked for a company that hosted its own source code control system. For us, as a small team, i…

Dunno, I got blocked from my work SaaS hosted gitlab for about a month by cloudflare. Nobody at gitlab or cf helped. I only figured it myself after about 4 hours of research, that it was caused by some disabled (by me years ago) web tracking APIs no-one should have hard dependence on.

I certainly would not have this problem on self hosted instance, because it would not be behind CF. I'm sure I'd have other problems though. :)

All software is crap. You can be either spending time fixing it yourself, or spending time begging online for fixes/help from some SaaS company/community with resolution time in months, somtimes, all that while you may not be able to use it fully.

Also with SaaS it will be constantly shifting under you. Things will be moved around, restyled, iconized, popupized, etc. This doesn't help productivity either. With self-hosting, you can at least avoid upgrading, if you dislike this kind of thing. Or choose FOSS software that values UX permanency/stability, which seems to be really hard ask from SaaS business.

Re: GitHub incident 2022-03-23

#104

Post by whimsicalism » Wed, Mar 23, 2022, 4:46 PM UTC

shepardrtc wrote: post #79

blueplanet200 wrote: post #20

I hope they figure out what’s going on every morning. Heard from inside they don’t know why the db dies everyday but restarting it fixes it.

IIS Server had/has a memory leak in worker threads that many years ago always forced us to restart the server every few days. Starting in 6.0, they added worker thread recycling and made it a mandatory to choose a time period for every thread to be recycled. Why fix the error when you can just restart the service?

I doubt they use IIS

Re: GitHub incident 2022-03-23

#105

Post by copperx » Wed, Mar 23, 2022, 4:47 PM UTC

encryptluks2 wrote: post #73

jmartens wrote: post #67

Earlier quoted context omitted.

How can we solve this as customers, or push the vendor to do better?

Stop being a customer of crappy vendors

What cloud provider does better status pages than AWS?

Re: GitHub incident 2022-03-23

#106

Post by mrkramer » Wed, Mar 23, 2022, 4:47 PM UTC

belter wrote: post #39

nimbius wrote: post #16

https://www.githubstatus.com/history 21 incident outages in just 3 months. At this rate the benefits of running your own gitea or gitlab are starting to become competitive.

Excluding ones reported as [Errors], [Scheduled] or [Notifications] 2019 -> 39 Incidents 2020 -> 67 Incidents 2021 -> 86 Incidents 2022 -> 20 Incidents so far Edit: Using Linear Regression...Prediction for total end 2022: 111 Incidents.

One would thought when they got acquired by Microsoft that the number of incidents would go down considering all resources Microsoft would provide but no.

Re: GitHub incident 2022-03-23

#107

Post by frjalex » Wed, Mar 23, 2022, 4:47 PM UTC

Looking at the "GitHub" prefix in the title, I was half-expecting this to point to a report explaining the outage a week ago... But rest assured, it is a new outage!

Re: GitHub incident 2022-03-23

#108

Post by TheRealPomax » Wed, Mar 23, 2022, 4:48 PM UTC

ejb999 wrote: post #84

belter wrote: post #39

Earlier quoted context omitted.

Excluding ones reported as [Errors], [Scheduled] or [Notifications] 2019 -> 39 Incidents 2020 -> 67 Incidents 2021 -> 86 Incidents 2022 -> 20 Incidents so far Edit: Using Linear Regression...Prediction for total end 2022: 111 Incidents.

thats not the kind of progression you like to see - that is, error rates increasing over time instead of decreasing.

Only if you believe those numbers mean anything. What are the errors for? Github has been adding lots of features and subproducts over the years, becoming a bigger and bigger platform as a result. What you want is the error-per-component, which may very well have actually gone down, with error spikes coming from "when github adds a completely new feature and it goes through a slew of incidents in its first year". The bigger the feature, the more incidents.

Without more detailed numbers, there's literally no conclusion to draw here.

Re: GitHub incident 2022-03-23

#109

Post by pid-1 » Wed, Mar 23, 2022, 4:49 PM UTC

Melatonic wrote: post #98

nimbius wrote: post #16

https://www.githubstatus.com/history 21 incident outages in just 3 months. At this rate the benefits of running your own gitea or gitlab are starting to become competitive.

The marketing for building your own new "private cloud" will begin soon I am sure :-D

You just made a few Openstack consultants raise from their graves.

Re: GitHub incident 2022-03-23

#110

Post by sonicggg » Wed, Mar 23, 2022, 4:49 PM UTC

nimbius wrote: post #16

https://www.githubstatus.com/history 21 incident outages in just 3 months. At this rate the benefits of running your own gitea or gitlab are starting to become competitive.

Come on, don't be so dramatic. This is not a 911 call center, people will survive these minor outages.