Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

GitHub incident 2022-03-23

githubstatus.com

Post reply on HN
Print view View original

61–70 of 206 posts

Re: GitHub incident 2022-03-23

#61

Post by nightpool »

gjulianm wrote: post #32
edgyquant wrote: post #22

Earlier quoted context omitted.

I’m not sure at what organization that is true. My company lives out of GitHub and Jira and I’ve hardly noticed the three month surge. GitHub would have to do a lot worse to get many companies to want to host their own services. This is the argument people have said about the cloud from day one. People want to know it isn’t their problem, that makes cloud computing (and things like GitHub) worth their weight in gold.…

At my organization it's always been true. Setting up GitLab is fairly easy, in my company we do it and it's cheap (on-prem hosting is basically zero, and we had the IPs/domains already) and it hasn't given us too many headaches. I think last time I had to do something was maybe a few months ago when I restarted it so that it picked up the updated SSL certificate.

Self-hosted GitLab got a good callout yesterday from Microsoft, it appears to be a favorite of LAPSUS$: https://www.microsoft.com/security/blog/2022/03/22/dev-0537-...

Self-hosting always increases the operational burden of making sure your systems are secure. Maybe you have the engineering resources to spend on patching everything immediately and conducting in-house pen tests, but for most companies it's much, much more secure to let the software's developers host it as well.

Re: GitHub incident 2022-03-23

#62

Post by mhitza »

iBotPeaches wrote: post #9

It seems like we haven't had a non-robot status update on the status page in days since this what seems like daily occurrence. I figure at this point we'd get something of why this is happening. I also don't appreciate our builds freezing, unable to be cancelled and then eating up hundreds of minutes.

I suggest you add the timeout-minute property on the job/step, so even if the web interface isn't responsive the job times out eventually. Saves you from spending time emailing support about consumed minutes.

Of course, assuming that a future bug won't affect the timeout-minute itself.

Re: GitHub incident 2022-03-23

#63

Post by toastal »

And to think Git can easily be decentralized. I wonder if the community could fork GitHub to fix it. Oh, it's not open source. Devs must be too busy working on more 'social' features like "For You (Beta)" to milk the attention economy.

Re: GitHub incident 2022-03-23

#65

Post by jmartens »

edgyquant wrote: post #22
nimbius wrote: post #16

https://www.githubstatus.com/history 21 incident outages in just 3 months. At this rate the benefits of running your own gitea or gitlab are starting to become competitive.

I’m not sure at what organization that is true. My company lives out of GitHub and Jira and I’ve hardly noticed the three month surge. GitHub would have to do a lot worse to get many companies to want to host their own services. This is the argument people have said about the cloud from day one. People want to know it isn’t their problem, that makes cloud computing (and things like GitHub) worth their weight in gold.…

My company monitors the functionality, performance and availability of apps like Github, and we have certainly noticed the increase in issues lately.

Re: GitHub incident 2022-03-23

#66

Post by gaoshan »

cube00 wrote: post #30
blueplanet200 wrote: post #20

I hope they figure out what’s going on every morning. Heard from inside they don’t know why the db dies everyday but restarting it fixes it.

Break out the early morning restart cron job.

Here you go, Github:

0 4 * * * /etc/init.d/postgresql restart

I'll take an architect position as compensation, but only if there is equity.

Re: GitHub incident 2022-03-23

#67

Post by jmartens »

mrkurt wrote: post #55
nimbius wrote: post #16

https://www.githubstatus.com/history 21 incident outages in just 3 months. At this rate the benefits of running your own gitea or gitlab are starting to become competitive.

If you want companies to be honest on their status pages (I do!), you can't just count incidents like that. Status pages can be an amazing place to communicate all kinds of problems. Most issues have a relatively narrow impact, but the impacted people _still_ benefit from seeing them listed.

How can we solve this as customers, or push the vendor to do better?

Re: GitHub incident 2022-03-23

#69

Post by mrkurt »

jmartens wrote: post #67
mrkurt wrote: post #55

Earlier quoted context omitted.

If you want companies to be honest on their status pages (I do!), you can't just count incidents like that. Status pages can be an amazing place to communicate all kinds of problems. Most issues have a relatively narrow impact, but the impacted people _still_ benefit from seeing them listed.

How can we solve this as customers, or push the vendor to do better?

Use vendors who do a good job communicating status, basically. I don't think you can change AWS behavior. But if you find a hosting company who does an amazing job with their status updates, put some apps there (_my_ company does an ok job with status page updates, we're getting better, it's not amazing yet).

Re: GitHub incident 2022-03-23

#70

Post by [deleted] »

adamsmith143 wrote: post #41
nimbius wrote: post #16

https://www.githubstatus.com/history 21 incident outages in just 3 months. At this rate the benefits of running your own gitea or gitlab are starting to become competitive.

Is it really though? Are engineers committing so frequently that they can't make it through a few hours without Github?

[deleted]
Post reply on HN