I think my biggest nightmare is SO, GH, and HN all being down at the same time. Edit: A nightmare is an exaggeration. But it would slow down work.
Could always just RTFM if SO goes down ️
GitHub is degraded/down
91–100 of 171 posts
Re: GitHub is degraded/down
#92Earlier quoted context omitted.
They've been pushing lots of features in the platform so not surprising that it's a bit unstable now.
In an ideal world, pushing new features would have no impact on stable mature features like browsing files, comment threads, etc
Internal to Amazon we consider that about 80% of issues/outages/etc are due to changes. This may sound "duh" but this is over 10k plus investigations.
Much if the work is just minimizing the impact of this changes by finding them before customers do.
This includes things like unit, integration testing, canaries, cellular/ zonal / regional deploys, auto rollbacks, multi-hour bakes, auto load tests, and much much monitoring. Not to mention cross team code reviews, game days, ops reviews.
Re: GitHub is degraded/down
#93Earlier quoted context omitted.
If you're looking for somewhere else, SourceHut has had no unplanned outages in 2020, despite being kept online by an army of one. The software and infrastructure is just more resilient and better maintained. Our ops guide is available here: https://man.sr.ht/ops/ It's also the highest performance software forge by objective measures: https://forgeperf.org/ Full disclosure: I am the founder of SourceHut.
Thanks for being so transparent about your operations. Maybe this is somewhere in the manual and I missed it, but do you have some way of automating the configuration of your hosts and VMs? For example, do you use something like Ansible?
No, I provision them manually. Being based on Alpine Linux makes this less time-consuming and more deterministic. At some point I might invest in something completely automated, but right now the manual approach is simpler - and if it's not broke, don't fix it.
Re: GitHub is degraded/down
#94Is it just me, or has Github's quality of service been continually degrading over the past several months? What is going on internally? Is this because of the Microsoft acquisition? Increased usage? An internal transition to Azure? ...is it time to move away from Github?
May it has more to do with their changes in pricing and a surge in uptake of customers?
Re: GitHub is degraded/down
#95Earlier quoted context omitted.
Github is as much or more a collaboration tool as a Git upstream. The Git part is easy to distribute. The collaboration tool, not so much. And to anticipate the likely next argument - no, mailing lists are not a better tool for collaboration. They are distributed, sure. In so saying, I have exhausted the list of their virtues.
The "we deploy production directly from Github" pathology is the one I'm talking about. Not being able to "collaborate" for a brief outage is fine. Not being able to deploy code isn't.
As a side note, gotta say, scare-quoting "collaborate" seems like a pretty weird flex in a time when everybody's stuck working from home whether they prefer it that way or not. Maybe you prefer to work entirely within a silo, rarely dealing with colleagues in the course of your day-to-day. Most don't, nor should they. Most kinds of work, even the kinds of work that we do, really aren't better done that way.
I worked in a silo for years, not so much by choice as by virtue of the fact that my company at that time could not afford another engineer of similar capability. I got by well enough, thanks in no small part to a youthful talent for improvisation, and the fact that things were simpler then. I wouldn't go back to that life if you paid me. But even if I would, I couldn't. Times change, and the time of the engineer as hermit has largely passed. There are things about it I miss, sure. But I don't miss them so much that I want to imperil my ability to keep earning a living by ignoring that the world has moved on.
edit: Of course, this ignores the difference between modern dev work, and modern sysadmin (as opposed to devops) work. It probably doesn't make much sense to analyze the latter as though it were the former. Or vice versa.
Re: GitHub is degraded/down
#96Is it just me, or has Github's quality of service been continually degrading over the past several months? What is going on internally? Is this because of the Microsoft acquisition? Increased usage? An internal transition to Azure? ...is it time to move away from Github?
My guess is opening their paying offering for free and also Github actions which is CPU-intensive and does much more than the traditional CI/CD tools.
Ideally GitHub Actions would be completely independent of GitHub's core services/servers (e.g. how Travis CI, Circle CI, etc. works), but that seems like it may not be the case.
Also, I'm still anticipating a fuller report on the database issues they mentioned have been the root cause of many outages over the past few months.
Re: GitHub is degraded/down
#97Earlier quoted context omitted.
Yes, and so this is actually a thing that bugs me, because I use Github every day. Over the past several months, there have been numerous days in which I've had problems (`You can not comment at this time`, 500s, etc.) and no corresponding status report. It seems like the historical uptime page paints a far rosier picture than I am actually experiencing.
It's almost as if keeping a complex service like Github online and available to millions of users is hard.
Building skyscrapers is hard. Does that make it OK for them to fall down regularly?
Re: GitHub is degraded/down
#98Is it just me, or has Github's quality of service been continually degrading over the past several months? What is going on internally? Is this because of the Microsoft acquisition? Increased usage? An internal transition to Azure? ...is it time to move away from Github?
May it has more to do with their changes in pricing and a surge in uptake of customers?
That would point to Azure hosting. Anyone notice a similiar pattern?
Re: GitHub is degraded/down
#99At my organization, we saw an uptick in timeouts spanning vendors that started at the same time - 6:55am PST. Makes me think there's an internet wide event occurring right now
Which providers are you seeing issues with? I'm curious if we can corroborate.
I was referring to WorldPay and Adyen (payment providers), but also saw issues directly w Github.
Re: GitHub is degraded/down
#100Why not make two versions of GitHub, one free and one paid, having professional-level uptime and support? Is that the idea behind GitHub Enterprise ( https://github.com/enterprise )?
Yes. Enterprise has an SLA (when GitHub hosts it) and an option to self-host.
Oh okay, I didn't know GitHub had an option to host Enterprise. (It's way overbudget for me right now, but just curious.)