Live data from Hacker News

GitHub incident 2022-03-23

githubstatus.com

161–170 of 206 posts

Re: GitHub incident 2022-03-23

#161
post #151
post #136

Earlier quoted context omitted.

The number of incidents isn't so much of a problem as the amount of downtime is. That would be more interesting to see.

GitHub Availability Report [1] Service Downtime Core Services Only - Cumulative per Month ( Some months with more than one outage) Jan 2021: 3 hours 53 min Feb 2021: 1 hour 42 min Mar 2021: 4 hours 10 min Apr 2021: 2 hours 20 min May 2021: 10 hours 34 min Jun 2021: 0 min Jul 2021: 0 min Aug 2021: 4 hours 23 min Sep 2021: 0 min Oct 2021: 1 hour 36 min Nov 2021: 2 hours 50 min Dec 2021: 0 min Jan 2022: 26 min Feb 2022:…

So, if my math is right (for 2021 only): 1888 min / 525,600 min = 99.64% uptime.

If it was more like 99.80+ I think I would be like "meh", but honestly for the price you pay that's not terrible. Still, for a company at the Microsoft level, it should be 99.80 at least.

Re: GitHub incident 2022-03-23

#162

Earlier quoted context omitted.

I think we pay about $10 per developer per month for github, and with about 1000 developers I would love that hourly rate.

...What? $10 x 1000 = $10k / month. $10k x 12 = $120k. That is a new grad software engineer salary in any US city. You'd pay more than that for a single dev with the devops and security experience to keep GHE running and patched for 1000 devs.

There are a lot of problems with this from the business angle:

(1) An engineer getting paid 120k doesn't "cost" 120k, probably >150k with federal taxes, health insurance, benefits, and so on. Not including the cost to recruit, interview, and train said person.

(2) I don't know of many 1,000 person companies that would trust a new grad software engineer with no experience to manage critical infrastructure.

(3) You need N engineers to manage said service, because what happens when your one engineer gets sick, takes PTO, or quits for some reason? You also need a manager for said engineer(s).

(4) You now need to secure an internal service you never did before, so expect to have to hire external security consultants or re-allocate security engineers, since it's high risk.

(5) Github is FedRAMP compliant, SOC1 and SOC2 compliant and GDPR compliant. If you or your customers need any of those things, expect to hire external auditors on a recurring basis to validate your home-grown solution meets those requirements.

I hate to make these points because I'm a big believer in the scrappy startup mentality, but if you want to do things right, in the context of a large enterprise that is accountable to a lot of people, expect a project like this to cost $1MM per year minimum, and it probably won't reach parity with a cloud offering in terms of reliability, multi-region performance, proper backups, and so on. This is why Github can charge ~$200 per user (Or $200k per year for 1,000 seats) and still come away looking like a bargain.

Re: GitHub incident 2022-03-23

#164
post #39
post #16

https://www.githubstatus.com/history 21 incident outages in just 3 months. At this rate the benefits of running your own gitea or gitlab are starting to become competitive.

Excluding ones reported as [Errors], [Scheduled] or [Notifications] 2019 -> 39 Incidents 2020 -> 67 Incidents 2021 -> 86 Incidents 2022 -> 20 Incidents so far Edit: Using Linear Regression...Prediction for total end 2022: 111 Incidents.

Based on the same interpolation, github will reach one incident per day by 2032.

Re: GitHub incident 2022-03-23

#165
post #89
post #66

Earlier quoted context omitted.

Here you go, Github: 0 4 * * * /etc/init.d/postgresql restart I'll take an architect position as compensation, but only if there is equity.

GitHub uses MySQL primarily though.

MySQL also has a restart command! I'll take my rsus now ty.

Re: GitHub incident 2022-03-23

#166
post #44

Earlier quoted context omitted.

I’d say it depends, I run my own on prem server and gitlab was a PITA. Too many moving parts, updating took too much of my time, and I never felt “safe”. Moving to gitea solved all of those issues for me (thus far), now I’m looking into adding other stuff like CI through Drone.

Did you consider woodpecker instead of drone? It's basically an evolved fork of the OSS version. https://woodpecker-ci.org/

Didn’t even know about it. I’ll check it out.

Thanks!

Re: GitHub incident 2022-03-23

#167

Earlier quoted context omitted.

They are green for good 15 minutes from first moment i see problems, not the first time, it happens actually quite often. Maybe that's the time they need to confirm/cross check/write status update, don't know.

While quicker reporting would be better, 15 minutes is anecdotally a lot better than I see from most other services where their status pages will report all-clear hours into full outages.

Yeah, I'm legit impressed with a 15 minute time here.

Re: GitHub incident 2022-03-23

#168

I hope they figure out what’s going on every morning. Heard from inside they don’t know why the db dies everyday but restarting it fixes it.

Yuck. Honestly, restarting a database to fix a major outage sounds like "we have no idea what we're doing"

Re: GitHub incident 2022-03-23

#170

Earlier quoted context omitted.

I doubt they use IIS

MSer here, yes we do… for some things

For GitHub? It seems unbelievable that they would use IIS pre-purchase and why in the world would you mix in a second web server for post-purchase enhancements.
Post reply on HN