Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

GitHub incident 2022-03-23

githubstatus.com

Post reply on HN
Print view View original

91–100 of 206 posts

Re: GitHub incident 2022-03-23

#91

Post by SkyPuncher »

edgyquant wrote: post #22
nimbius wrote: post #16

https://www.githubstatus.com/history 21 incident outages in just 3 months. At this rate the benefits of running your own gitea or gitlab are starting to become competitive.

I’m not sure at what organization that is true. My company lives out of GitHub and Jira and I’ve hardly noticed the three month surge. GitHub would have to do a lot worse to get many companies to want to host their own services. This is the argument people have said about the cloud from day one. People want to know it isn’t their problem, that makes cloud computing (and things like GitHub) worth their weight in gold.…

> I’m not sure at what organization that is true. My company lives out of GitHub and Jira and I’ve hardly noticed the three month surge

These have been minor inconveniences for us - at worst. Most of the time it simply means people jump to something else then come back later in the day.

Failing tests and PR feedback cycles are more of a blocker to our team than these outages.

Re: GitHub incident 2022-03-23

#92

Post by temp8964 »

nightpool wrote: post #61
gjulianm wrote: post #32

Earlier quoted context omitted.

At my organization it's always been true. Setting up GitLab is fairly easy, in my company we do it and it's cheap (on-prem hosting is basically zero, and we had the IPs/domains already) and it hasn't given us too many headaches. I think last time I had to do something was maybe a few months ago when I restarted it so that it picked up the updated SSL certificate.

Self-hosted GitLab got a good callout yesterday from Microsoft, it appears to be a favorite of LAPSUS$: https://www.microsoft.com/security/blog/2022/03/22/dev-0537-... Self-hosting always increases the operational burden of making sure your systems are secure. Maybe you have the engineering resources to spend on patching everything immediately and conducting in-house pen tests, but for most companies it's much, much…

Not necessarily. Self-hosted services are protected by company firewall / VPN. They can setup very restrictive network access. They don't have the same level of risks as public services like GitHub or GitLab.

Re: GitHub incident 2022-03-23

#93

Post by djbusby »

shepardrtc wrote: post #79
blueplanet200 wrote: post #20

I hope they figure out what’s going on every morning. Heard from inside they don’t know why the db dies everyday but restarting it fixes it.

IIS Server had/has a memory leak in worker threads that many years ago always forced us to restart the server every few days. Starting in 6.0, they added worker thread recycling and made it a mandatory to choose a time period for every thread to be recycled. Why fix the error when you can just restart the service?

Apache prefork had that since forever. Seems just a garbage collect type pattern.

Re: GitHub incident 2022-03-23

#94

Post by edgyquant »

jmartens wrote: post #65
edgyquant wrote: post #22

Earlier quoted context omitted.

I’m not sure at what organization that is true. My company lives out of GitHub and Jira and I’ve hardly noticed the three month surge. GitHub would have to do a lot worse to get many companies to want to host their own services. This is the argument people have said about the cloud from day one. People want to know it isn’t their problem, that makes cloud computing (and things like GitHub) worth their weight in gold.…

My company monitors the functionality, performance and availability of apps like Github, and we have certainly noticed the increase in issues lately.

We were actually talking about implementing this last week. Not for GitHub but for slack as it seems to have issues once a month or so.

Re: GitHub incident 2022-03-23

#95

Post by Melatonic »

nightpool wrote: post #61
gjulianm wrote: post #32

Earlier quoted context omitted.

At my organization it's always been true. Setting up GitLab is fairly easy, in my company we do it and it's cheap (on-prem hosting is basically zero, and we had the IPs/domains already) and it hasn't given us too many headaches. I think last time I had to do something was maybe a few months ago when I restarted it so that it picked up the updated SSL certificate.

Self-hosted GitLab got a good callout yesterday from Microsoft, it appears to be a favorite of LAPSUS$: https://www.microsoft.com/security/blog/2022/03/22/dev-0537-... Self-hosting always increases the operational burden of making sure your systems are secure. Maybe you have the engineering resources to spend on patching everything immediately and conducting in-house pen tests, but for most companies it's much, much…

Except that the software developers hosting is also a much, much bigger target and you generally do not have any real control over how often they are patching either.

Re: GitHub incident 2022-03-23

#97

Post by Someone »

rvz wrote: post #24
nimbius wrote: post #16

https://www.githubstatus.com/history 21 incident outages in just 3 months. At this rate the benefits of running your own gitea or gitlab are starting to become competitive.

Well, I think I have said that since 2020 [0] and it is self-evident that you are better off self-hosting your own Git repo. If you can host a website you can do it. If GNOME, ReactOS, Wireguard, Linux Kernel Project, Mozilla, etc can do it, so can you. Or even use it as a backup / failsafe just in case. But going 'all in' on GitHub just doesn't make any sense anymore. [0] https://hn.algolia.com/?dateRange=all&page=1…

But who can host a website? I would be wary of hosting something that isn’t a 100% static site, out of fear of the amount of attention maintenance would take.

Also, quite a few of the non-profits behind the projects you mentioned have multi-million dollar budgets that they can use to administer their git instance, if needed. I don’t think “if they can do it, you can” is a strong argument for those.

Re: GitHub incident 2022-03-23

#98

Post by Melatonic »

nimbius wrote: post #16

https://www.githubstatus.com/history 21 incident outages in just 3 months. At this rate the benefits of running your own gitea or gitlab are starting to become competitive.

The marketing for building your own new "private cloud" will begin soon I am sure :-D

Re: GitHub incident 2022-03-23

#99

Post by tjoff »

kleebeesh wrote: post #88
capitol_ wrote: post #82

Earlier quoted context omitted.

I think we pay about $10 per developer per month for github, and with about 1000 developers I would love that hourly rate.

...What? $10 x 1000 = $10k / month. $10k x 12 = $120k. That is a new grad software engineer salary in any US city. You'd pay more than that for a single dev with the devops and security experience to keep GHE running and patched for 1000 devs.

Well, considering you'd likely spend an average of 5 minutes per day doing it I wouldn't mind it.

Re: GitHub incident 2022-03-23

#100

Post by cortesoft »

kleebeesh wrote: post #88
capitol_ wrote: post #82

Earlier quoted context omitted.

I think we pay about $10 per developer per month for github, and with about 1000 developers I would love that hourly rate.

...What? $10 x 1000 = $10k / month. $10k x 12 = $120k. That is a new grad software engineer salary in any US city. You'd pay more than that for a single dev with the devops and security experience to keep GHE running and patched for 1000 devs.

The person was replying to a comment saying they spend more on a SINGLE HOUR of a dev's time than the monthly GH bill, which is not true for an org of more than 20 people or so (depending on hourly rate).
Post reply on HN