https://status.github.com shows > 18:19 CDT Major service outage.
What's CDT?
Central Daylight Time
GitHub was down
181–190 of 197 posts
Re: GitHub was down
#182Earlier quoted context omitted.
It's obviously possible to manage very large projects stored in Git repos with no dependency on a site like GitHub at all. Just look at what Git was designed for, after all. However, I'd say one of the biggest practical problems with most of the popular DVCS tools is still that they don't also have a simple, out-of-the-box way to set up a centralised repo. As you say, using Git+SSH works up to a point, but it's still…
It depends on what your deployment system is; it's not always trivial to just change the target git origin to a new server. For someone who knows what they're doing and has the time to focus on doing it, it won't be too hard, but Murphy's Law gets in the way.
It depends on what your deployment system is; it's not always trivial to just change the target git origin to a new server.
This seems to be a fundamental weakness in whatever deployment system is used, then. Relying on a single point of failure outside of your organisation's own control for anything critical is always risky, and in this case it seems to be entirely avoidable.
Re: GitHub was down
#183Earlier quoted context omitted.
> But I'll posit that you never actually had control, only the illusion of control. That's a weird way of phrasing it. When you run your own services and they break you have total control and do have the power to fix it. When you buy SaaS you are relying on someone else. You may very well have more reliability and uptime but you are nonetheless giving up control.
That's exactly the point. Sure you have "control" but what good is all that control if it takes you 4 hours to track down the source of a problem and fix it (for example). Github has 100s of engineers (with heavily specialized knowledge that you don't have btw) working to fix any problems, I'd bet on them over myself and (maybe a handful of engineers) everytime ... and I've run my own subversion/git servers before. S…
Your post gets to the false tradeoff that turned "devops" from a push for better interdisciplinary collaboration into a "2 jobs, 1 paycheck" role.
Software teams are often pathologically unable to create a deliverable that is scrutable and manageable by an operations audience. Building a product that can stand on it's own two feat without constant minding is work.
For all the testing dogma that files around, constant delivery has resulted in systems that are more brittle than ever. If someone with all the institutional knowledge is always there to catch the system when it fails, why bother making it easy to investigate failures? Why bother with analysis and building in fault tolerance when someone can worry about a long term solution for that failure mode when it causes them to get paged at 3 am?
So it becomes easy to say minders are necessary, and they must be developers. Business incentives mean that answer isn't always scrutinized as heavily as it should be, because it means spreading maintenance costs over time instead of an upfront investment in resilience and maintainability. Refusing to provide a toolset and manual for maintenance means some nobody with access to google can't fix 99% of the problems that could occur. That way the magic black box creators can ensure they're the ones getting paid to do it.
We don't have or need Windows engineers, kernel engineers, or Cisco engineers on call. Nor do we have nginx, postgres, cpython, exim, apache, php, mysql, Active Directory, Exchange or Office engineers on call. We use weird enterprise software from companies that have gone out of business, we don't have high-priority support contracts with many of the rest.
Software that needs minding from the developers is just bad software. That's technical debt they took on to get the product out the door.
Re: GitHub was down
#184Earlier quoted context omitted.
Your question seems to indicate that you have a runtime dependency on github. If true, this is a problem no matter where your git repo resides. You need to architect a solution wherein you do not have a runtime dependency, especially if it is a single point of failure, or accept that your reliability will never be better than the sum of your services' reli abilities (there was an article on the "calculus of availabil…
Is it still considered a runtime dependency if all you do is merge code and deploy while relying on GitHub? I assume most apps will not go down if GitHub goes down, but their ability to move code to production gets stalled.
> Is it still considered a runtime dependency if all you do is merge code and deploy while relying on GitHub?
It's not a runtime dependency, it's a stupidity to rely on random repositories from the internets when building and deploying (remember the left-pad farca that halted work for half of the web programmers). You should always have your local copy of necessary sources or use repositories that can trivially be swapped with some other mirror.
We used to have the infrastructure for those things (in the form of binary and source package repositories), but it was mainly used by sysadmins, and programmers usually have an alergy for such infrastructure.
Re: GitHub was down
#185Earlier quoted context omitted.
But in theory in a pinch you could spin up a git server somewhere publicly accessible and switch your site to pull from there, and push to there yourself.
Not if you're using more than git from github, for example downloading releases as zip files for deployment. There are probably a lot of other tempting features to put on your deployment critical path but I'm not in an environment that uses github so I've forgotten.
Yeah definitely. I was just thinking about the basic use case where you deploy from it and that's it.
Re: GitHub was down
#186Earlier quoted context omitted.
But in theory in a pinch you could spin up a git server somewhere publicly accessible and switch your site to pull from there, and push to there yourself.
Not if you're using more than git from github, for example downloading releases as zip files for deployment. There are probably a lot of other tempting features to put on your deployment critical path but I'm not in an environment that uses github so I've forgotten.
Webhooks are a big one: rather than polling a git server for changes it is very easy for CI/CD systems to rely on GitHub's HTTP webhooks in their critical path to kick things off on push/PR/branch/merge. Especially because GitHub does a great job of populating its webhooks with tons of useful information about the event that's tough to replicate with just git post-hooks.
Re: GitHub was down
#187Earlier quoted context omitted.
At this point Github is a national security risk. The amount of passwords, keys, hashes, code, data uploaded to that system is massive. I am sure that government agencies, contractors, multinational defense departments, etc are dumping things on there hourly. Will it get hacked ? Get your popcorn out. It will happen.
Is GitHub even allowed for govt contractor use? I'd be surprised, but don't know.
For some projects, yes. The biggest well known US examples are the 18F open source efforts and http://code.gov / http://code.mil open source efforts, most of which are public, open source on GitHub.
Re: GitHub was down
#188Earlier quoted context omitted.
Exactly this. It's painful to realize that Git was invented to solve this very problem, but so many use GitHub as their source control server just like CVS and all the other old source control systems.
Then what's the alternative? Push and pull directly from other users who have access to the repository? What if that user is currently offline? There are huge benefits to having a main centralized repo. If you need redundancy for some reason, it's fairly trivial to mirror the repository somewhere else.
I always thought that an IM-style tool with online/offline presence information of remote repositories could be a fascinating tool for DVCS source control repositories.
Tried to prototype on XMPP at one point in college but never got very far in the prototype.
(That was back when I was a heavy darcs proponent too and had other crazy things like automated consensus branching as a great distributed way to coordinate efforts on such a distributed project, which would work with the darcs push/pull/merge model but not so well in git's.)
Re: GitHub was down
#189Earlier quoted context omitted.
I kind of see your point but it's the wrong phrase. It's not a runtime dependency in a technical sense.
see now you're just splitting hairs and getting into philosophy. if your website will be stale tomorrow because you can't update it today, then that is a runtime dependency imo.
Why it is down tomorrow?
(Hint: sans hug of death, because a runtime dependency brought it down.)
Re: GitHub was down
#190Earlier quoted context omitted.
Gitea sadly has some issues (for example, the "Public Activity" tab on user profiles 500's atm: https://try.gitea.io/crackedmind?tab=activity ; it throws 500's everywhere once you throw on a modern Xeon more than 30 users/second at it, because it doesn't cache database accesses, at all, etc).
Thanks for mentioning that. It seems like this bug wasn't reported yet, so I have filed a bug report.
Also, I don’t seem to be able to properly run it in docker (I’m trying to deploy it on kubernetes), do you, by random chance, have experience with deploying gitea?