Live data from Hacker News

GitHub incident 2022-03-23

githubstatus.com

181–190 of 206 posts

Re: GitHub incident 2022-03-23

#181

Earlier quoted context omitted.

MSer here, yes we do… for some things

If GH is around the same level of integration with Microsoft as my employer, which is another Microsoft acquisition, I don't really believe you have a ton of insight into GH processes.

I dated a girl at GitHub for awhile last year who said they weren’t even completely off of AWS yet and she liked how they didn’t seem like working for Microsoft. Maybe this has changed though.

Re: GitHub incident 2022-03-23

#183
post #179

Earlier quoted context omitted.

Yuck. Honestly, restarting a database to fix a major outage sounds like "we have no idea what we're doing"

Would you rather it stay down while they spend a day debugging it?

If that means it won't be down every morning in my time zone then yes.

Re: GitHub incident 2022-03-23

#184

I hope they figure out what’s going on every morning. Heard from inside they don’t know why the db dies everyday but restarting it fixes it.

Yuck. Honestly, restarting a database to fix a major outage sounds like "we have no idea what we're doing"

They could use multiple writer hosts and rollover the restarts. MySQL has had GTIDs since 5.6 and replication groups rather than writer-replicas since some 5.7.x version.

Re: GitHub incident 2022-03-23

#185
post #43

I hope they figure out what’s going on every morning. Heard from inside they don’t know why the db dies everyday but restarting it fixes it.

What's "the db"? It sounds like something of small to medium scale if you can just restart it like that. In any case, why not just relocate some vendor engineers on site for a bit? Or, better, why does the vendor not have a small presence in the corner? Sounds like whatever "the db" is it's probably some (objectively) small but very scary thing that's currently on fire and people are trying to figure out how to put i…

Sounds like it was a MySQL database:

https://github.blog/2022-03-23-an-update-on-recent-service-d...

Re: GitHub incident 2022-03-23

#186
post #119

Earlier quoted context omitted.

I don't recall ReactOS, or the creators of wireguard having 'multi million dollar budgets' . How is it that even projects like RedoxOS [0] are able to self-host on a GitLab instance using a subdomain, without giant budgets in the millions? You don't need a 'multi-million dollar budget' to self-host a git repo and may of these open-source projects have been doing so even before GitHub existed for years. Even if they d…

> You don't need a 'multi-million dollar budget' to self-host a git repo I never made that claim. The argument was “if X can do it, so can you”. I pointed out that _some_of_these_ (Mozilla, likely the most extreme of them, had over $400 million in revenues in 2020), are quite different from the typical ‘you’, invalidating that argument. As always, invalidating an argument doesn’t mean its conclusion is wrong.

> The argument was “if X can do it, so can you”.

So when are you going to question this user [0] and others here planning to do the same thing for not having a 'multi-million dollar budget' for self-hosting their own services then?

Since clearly according to you they 'can't do it', despite me saying 'if X can do it so can someone else'. Where 'X' can be even a toy project like RedoxOS, or a messenger project like GNU Ring hosted by themselves and accessed via a subdomain.

Seems like they and other lesser known and funded open-source projects are doing just fine like that for years.

[0] https://news.ycombinator.com/item?id=30780874

Re: GitHub incident 2022-03-23

#187
post #179

Earlier quoted context omitted.

Yuck. Honestly, restarting a database to fix a major outage sounds like "we have no idea what we're doing"

Would you rather it stay down while they spend a day debugging it?

As long as it's announced in advance so that users/customers can plan ahead, I don't see why not.

Re: GitHub incident 2022-03-23

#189
post #172

Earlier quoted context omitted.

It's also itil lingo, which predates sre.

It's NIMS - FEMA lingo, which predates ITIL. Which was developed in USFS wildland firefighting, which predates FEMA. It's incident management all the way down.

"The Simpsons already did it"

Re: GitHub incident 2022-03-23

#190

I hope they figure out what’s going on every morning. Heard from inside they don’t know why the db dies everyday but restarting it fixes it.

Yuck. Honestly, restarting a database to fix a major outage sounds like "we have no idea what we're doing"

It sounds like "they don't know why it's going down." I've worked with plenty super competent people that have taken time to root cause incidents.

Guide to incidents: Step 1: Stop the bleeding Step 2: Prevent it in the future

Doing Step 1 doesn't make you incompetent.

Post reply on HN