Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

GitHub incident 2022-03-23

githubstatus.com

Post reply on HN
Print view View original

201–206 of 206 posts

Re: GitHub incident 2022-03-23

#201

Post by TheRealPomax »

AlexandrB wrote: post #132
TheRealPomax wrote: post #125

Earlier quoted context omitted.

Every place I ever worked at understood that if you x3 the codebase/infra/interaction surface/etc, you can expect x3 errors. If the total number of errors don't go up as you grow you're doing amazing, and if they go down even though you're landing more and more code for more and more features and subproducts, you have a genuine miracle.

These features can't be rolled out incrementally to users? In this day and age it seems weird for a web app to do a global go-live with something before testing it with a smaller group first.

A "smaller group" on github's scale is still large enough to take down an entire sub product like actions, hooks, codegroups, etc.

Re: GitHub incident 2022-03-23

#202

Post by raffraffraff »

blueplanet200 wrote: post #190
raffraffraff wrote: post #168

Earlier quoted context omitted.

Yuck. Honestly, restarting a database to fix a major outage sounds like "we have no idea what we're doing"

It sounds like "they don't know why it's going down." I've worked with plenty super competent people that have taken time to root cause incidents. Guide to incidents: Step 1: Stop the bleeding Step 2: Prevent it in the future Doing Step 1 doesn't make you incompetent.

I'm not a DBA, and maybe you're not a DBA either, so this question goes to DBAs who may be reading: aren't you always better off killing the bad queries instead of rebooting the whole box, if that's an option? (ie: aside from times when the entire host is screwed, load per core is >50, metrics aren't getting out, you can't ssh in etc)

Re: GitHub incident 2022-03-23

#203

Post by rglullis »

aaaaaaaaata wrote: post #200
djbusby wrote: post #90

Earlier quoted context omitted.

GitHub/Lab are for more than just code repo

Can project management features not be made part of a dumb repo on the db side? (Spoiler: yes, and many projects have explored this — setup unfortunately has never been as easy as "we'll invite u to the gh, check ur email". Perhaps with decentralization push of web3/QR etc, we'll get over the hump.

I think that parent means also things like CI, release repository, PR review, etc.

These are not easily portable, but honestly is because of this lock-in that I prefer to use separate/independent tools. For my open source project [0], I am putting things on github and it is the link that give to most people, but in reality is just a mirror to the gitlab repository[1], which I use for CI and static page hosting, and the "project management" is done on Taiga [2]

  [0]: https://github.com/mushroomlabs/hub20
  [1]: https://gitlab.com/mushroomlabs/hub20/hub20
  [2]: https://tree.taiga.io/project/lullis-mushroomlabshub20

Re: GitHub incident 2022-03-23

#204

Post by Yuioup »

prepend wrote: post #170
throwra620 wrote: post #123

Earlier quoted context omitted.

MSer here, yes we do… for some things

For GitHub? It seems unbelievable that they would use IIS pre-purchase and why in the world would you mix in a second web server for post-purchase enhancements.

Why trade an open source solution with third rate garbage that is called IIS which runs on a sub-par desktop OS called Windows. I thought that Github was supposed to be independant.

Re: GitHub incident 2022-03-23

#206

Post by TheRealPomax »

sitzkrieg wrote: post #155
TheRealPomax wrote: post #102

Earlier quoted context omitted.

But how many of those actually affected you? For example, no amount of issues around codespaces or github packages would impact my professional use of github, so whether there are 21 or 5000 or those parts get permanently taken offline makes no difference in what I need out of the platform. How many core incidents? The part that affects whether you can even push to and pull from a repo, and access issues and PRs? Bec…

i could not even sso login so it was a bit more impactful than it sounds on the paper

Absolutely, but could you not log in every single time, or just "once this year so far"?
Post reply on HN