ya’ll do know Git is a distributed VCS right? it’s ok for the the remote to be offline.
Incident with GitHub Actions, API requests, Codespaces, Git operations, Issues
101–110 of 123 posts
Re: Incident with GitHub Actions, API requests, Codespaces, Git operations, Issues
#102Earlier quoted context omitted.
> The entire point of git is that it's decentralized, lol. No-one here is criticizing git itself. That is not the point. It is GitHub that is defeating the whole point of it all, once their hosted central server goes down. The majority of these projects went all in on GitHub, including using GitHub actions, npm packages, hosting their whole website, etc hence as soon as it goes down, they can't push or update anythin…
If you're not building some downtime into your model you're not being realistic. It's easy to point fingers but the reality is every product and company will experience unexpected downtime. It's an easy business decision for executives/buyers, pay a team of top engineers to home grow a durable product assuming it can even be done at extreme cost now and later or be okay with a couple of hours of downtime here and the…
> Every single project you listed uses Github as a mirror meaning when they go down internally, Github is the backup which from my perspective is a little ironic.
And? It is a read-only mirror. It just 'pulls' changes from the self-hosted copy. It can't be used for direct development for the maintainers. If the main official repository was on GitHub and that goes down, then everything will be down as well including (issues, pull requests, actions, etc). Then you will be totally reliant on GitHub for 'fix it'.
There is a reason why those same projects do not use GitHub as their main repository and tell you 'We don't accept issues or patches here'8. They have control over their issues trackers, review process and CIs and their projects won't halt due to GitHub's unpredictable and intermittent issues.
For those projects, GitHub is only* used as a read-only mirror for cloners, but useless for anyone to send patches, track issues, PRs, etc. which that is done on their self-hosted repositories and it has been like that for them for years.
Re: Incident with GitHub Actions, API requests, Codespaces, Git operations, Issues
#103Earlier quoted context omitted.
-f does not sound like a good idea to me in a script like that.
Also pretty much every usage of -f would be better off being --force-with-lease so you're less likely to accidentally clobber someone else's work. I have git fpush aliased to "push --force-with-lease" and try to spread the gospel when possible :)
Yeah, I learned it by using magit or vscode's other magit and they both default to --force-with-lease.
Re: Incident with GitHub Actions, API requests, Codespaces, Git operations, Issues
#104Does anybody else remember when GitHub's outage page used to have little graphs showing downtime? Eventually they took it down as their outages were just too often. GitHub has _always_ had terrible uptime. It's a great product - wish something would change but it seems cultural at this point.
They had massive problems with their main database cluster (MySQL). If you read through their engineering blog, most of the outages were related to their growth and the main database cluster. They moved workloads for some features to different clusters, but that's only to buy more time. Eventually they'll do proper shredding (by user or org I guess, not by feature) but that takes time.
Their engineering blog is full of articles about MySQL and the main "mysql1" database cluster, e.g. https://github.blog/2021-09-27-partitioning-githubs-relation...
Re: Incident with GitHub Actions, API requests, Codespaces, Git operations, Issues
#105Earlier quoted context omitted.
> It is GitHub that is defeating the whole point of it all, once their hosted central server goes down. server != service assuming its a distributed service vs one server for a multi-billion$ company also group of humans built this service, so its not gonna be perfect :shrug: companies that use such tools and in trust all the business process to a provided service and do consider an event like this is a blocker shoul…
> assuming its a distributed service vs one server for a multi-billion$ company also group of humans built this service, so its not gonna be perfect :shrug: Regardless of any of that, it still is proven to be unreliable. It is also not an excuse to go all in and risk being fully dependent on GitHub (and their services) and tolerate such downtimes and run to HN and complain about it each month. > companies that use su…
> Then I should see no-one being surprised or complaining
Oh agree 100%, this is the equivalent of the "reply-all email threads" and people responding to be remove or stop. I find it entertaining overall.
> Until the next time GitHub goes down again
Cheers
Re: Incident with GitHub Actions, API requests, Codespaces, Git operations, Issues
#106Earlier quoted context omitted.
The last time I checked, the only meaningful parts of GitHub that ran on Azure was/is Actions. Everything else is AWS.
> Everything else is AWS. Huh? As of at least 2017 GitHub was running their own data centers [1]. Any evidence that’s changed? Microsoft bought them in 2018, I can’t imagine they went to AWS after that. [1]: https://github.blog/2017-10-12-evolution-of-our-data-centers...
Ah sorry, yeah I wasn't being very accurate.
I was checking "what's on Azure", I didn't really follow up to check in detail the breakdown of the rest. I just saw quite a bit on AWS and assumed it all was.
Re: Incident with GitHub Actions, API requests, Codespaces, Git operations, Issues
#107Earlier quoted context omitted.
If you're not building some downtime into your model you're not being realistic. It's easy to point fingers but the reality is every product and company will experience unexpected downtime. It's an easy business decision for executives/buyers, pay a team of top engineers to home grow a durable product assuming it can even be done at extreme cost now and later or be okay with a couple of hours of downtime here and the…
> Every single project you listed uses Github as a mirror meaning when they go down internally, Github is the backup which from my perspective is a little ironic. And? It is a read-only mirror. It just 'pulls' changes from the self-hosted copy. It can't be used for direct development for the maintainers. If the main official repository was on GitHub and that goes down, then everything will be down as well including (…
It's a remote origin, once I clone and branch which I can do from a mirror, I can write and commit as much as I want to the repo, where I push the change up to is ultimately my decision assuming I have access. The point stands, these companies use Github to act as a mirror/backup for their project in the event of something like a disaster (e.g. datacenter fire).
There is no perfect solution and there never will be. Everything has associated cost. You're focused on the distribution of devops tooling, but that is only a fraction of the story. Many large companies have moved to Saas based products because they realize doing it themselves comes with significant cost. An hour or two of downtime is cheaper then a datacenter, equipment, bandwidth, licensing, and expertise to manage all of it.
It's a simple cost benefit analysis. You need to look at this issue through the lens of a business and not just an engineer would be my advise. Interestingly enough you can only point to OSS projects which rarely pay for tooling anyways.
Re: Incident with GitHub Actions, API requests, Codespaces, Git operations, Issues
#108This is why you should have your code on multiple remotes, i.e. Azure DevOps, Git labs, self hosted git server.
Re: Incident with GitHub Actions, API requests, Codespaces, Git operations, Issues
#109Earlier quoted context omitted.
Don't you guys have other features and stuff to work on locally? What is this "time to take a break when GitHub is down"? I'm saying this a bit tongue in cheek btw :)
All my features are part of 1 PR, the PR contains no code to avoid bugs, the features are in my head.
I can’t see any deficiencies in this approach.
Re: Incident with GitHub Actions, API requests, Codespaces, Git operations, Issues
#110Does anybody else remember when GitHub's outage page used to have little graphs showing downtime? Eventually they took it down as their outages were just too often. GitHub has _always_ had terrible uptime. It's a great product - wish something would change but it seems cultural at this point.
They had massive problems with their main database cluster (MySQL). If you read through their engineering blog, most of the outages were related to their growth and the main database cluster. They moved workloads for some features to different clusters, but that's only to buy more time. Eventually they'll do proper shredding (by user or org I guess, not by feature) but that takes time. Their engineering blog is full…
i've noticed this too .. the real head-scratcher is how a solid chunk of github's db & infra folks left to join a database startup, one of them even becoming its ceo!!
if they had made github db/infra super-stable before this, it would be a vote of confidence in their new company, but instead imho it is the opposite