My repos were coming up as 404s. I was wondering if I had been canned...
We were warned multi-deploys with big changes were incoming: "For lack of a better term, some big shit is coming at GitHub Universe." - Thomas Dohmke, CEO
We are investigating reports of degraded performance
61–70 of 152 posts
Re: We are investigating reports of degraded performance
#62I like how "Every request returns a 404" == "degraded performance"
At least it did return a response.
Re: We are investigating reports of degraded performance
#63My repos were coming up as 404s. I was wondering if I had been canned...
Not mutually exclusive - GitHub could be down AND you could be canned!
Re: We are investigating reports of degraded performance
#64Earlier quoted context omitted.
I realize you're talking about git, but I think it's also pretty important to have a place where users can submit issues and dev's can say "it's fixed in version XYZ", these are not features of git. I've been meaning to give radicle a try.
Fossil does this within the repo, maybe we need a way for Git to do so too.
... Or use Fossil?
Re: We are investigating reports of degraded performance
#65Earlier quoted context omitted.
- 'one more step away from a corporate-controlled internet" Downvote me to grey-world if you like, but I think everyone's crazy to put all their code infrastructure in the hands of fucking Microsoft . Especially literal free open-source software. Who do you think Microsoft is? What do you know of Microsoft's history and their core values (they're "embrace, extinguish & exsanguinate"). It's like giving fucking Sauron…
But, I like the stock.
Now imagine having your shares managed by a Microsoft bank — maybe in SharePoint. ;)
Re: We are investigating reports of degraded performance
#66Earlier quoted context omitted.
It should load faster if there's nothing to load
If nothing loads nothing gets cached and you can get in a cycle of very slow 404s. The current abomination I'm working on avoids this by caching the errors and serving them for several hours...
One of the worst outages I witnessed was due to negative DNS caching on the most outer router that took the company a few (3 or 5) working days to fix *after* issue was identified.
Re: We are investigating reports of degraded performance
#67Github is becoming too important. A routine backup system is needed, something like archive.org, that backs up important Github projects.
I know this a greybeard's fantasy and that most people working today were trained not to bother, but: important things should not have GitHub as a failure point . Hobby projects and today's work? Sure. Point straight at GitHub and hack away. And when it goes down, get yourself a coffee. But everything that's anywhere near production should have already pointed those github requests to a mirror or other tool in your o…
It's not that black and white. Where do you draw the line on what can and can't be a failure point?
My cloud provider is probably an acceptable point. If every AWS region goes down I'm not going to have a spare cloud provider.
What about an auth provider? Do I need a backup there?
What about CI, do I need multiple CI systems?
3rd party search services, realtime messaging services, the list goes on.
For 1% of systems, you need backups for all of these (or to not use anything external). The other 99%, building backups for every one of these systems is a losing business strategy.
Some of them sure, but which those are will vary based on the context. It's not as simple as having a backup for "every other kind of remotely distributed dependency."
Re: We are investigating reports of degraded performance
#68I need a macro template for those memes with Bart Simpson on the blackboard and make it say "I will always have a backup plan for third-party services". Seriously people: gitea exists. Gitlab self-hosted exists. Drone/Woodpecker CI exists. It's not that difficult to set up a project that does not depend on Github. I spent less time setting these up than the amount of down time that Github has had this year.
Problem is, this is one of 50 things that can go wrong for which you should have a robust backup plan for
It's amazing how many of these issues can be obviated by taking a step back at a given SAAS, asking can I self host this, then if you can and need redundancy to just buy two desktops and stick one in your place and another in your friends apartment in another town. With a lightweight static site, a modern desktop is probably more than powerful enough to deal with most any load you might realistically see for your given project. You are also extremely unlikely to have both of these desktops go down at once if they are on different local power grids and internet service providers, short of invasion of the continental U.S. perhaps.
Re: We are investigating reports of degraded performance
#69Github is becoming too important. A routine backup system is needed, something like archive.org, that backs up important Github projects.
I know this a greybeard's fantasy and that most people working today were trained not to bother, but: important things should not have GitHub as a failure point . Hobby projects and today's work? Sure. Point straight at GitHub and hack away. And when it goes down, get yourself a coffee. But everything that's anywhere near production should have already pointed those github requests to a mirror or other tool in your o…
Interesting perspective, disagree on one central tenant. Even though Github holds production code, it is not production. Built artifacts and the machines running those artifacts is production. When Github goes down, which it rarely does, it just means developers cant sync for a couple hours, no different than if someone works offline. The temptation to increase internal devops complexity should not be an automatic immune response when a service goes down, it comes with all sorts of hidden costs.
Re: We are investigating reports of degraded performance
#70[deleted]