(epistemic status: frustrated, irrational)
This is what happens when they decide that all the budget should be spent on AI stuff rather than solid infra and devops
GitHub: Git operation failures
251–260 of 335 posts
Re: GitHub: Git operation failures
#252Earlier quoted context omitted.
Curious whether you actually think this, or was it sarcasm?
It was sarcasm, but git itself is Decentralized VCS. Technically speaking, every git checkout is a repo of itself. GitHub doesn't stop me from having the entire repo history up to last pull, and I still can push either to the company backup server or my coworker directly. However, since we use github.com fore more than just a git hosting it is SPOF in most cases, and we treat it as a snow day.
Yep, agreed - Issues being down would be a bit of a killer.
Re: GitHub: Git operation failures
#253Earlier quoted context omitted.
You can make it as reliable as you want by hosting it on prem.
> as reliable as you want We self-host GitLab but the team owning it is having hard time scaling it. From my understanding talking to them, the design of gitaly makes it very hard to scale it beyond certain repo size and # of pushes per day (for reference: our repos are GBs in size, ~1M commits, hundreds of merges per day)
[deleted]
Re: GitHub: Git operation failures
#254Re: GitHub: Git operation failures
#255Earlier quoted context omitted.
As long as you didn't go all in on GitHub Actions. Like my company has.
Do you think you'd get better uptime with your own solution? I doubt it. It would just be at a different time.
Yes. I've quite literally run a self-hosted CI/CD solution, and yes, in terms of total availability, I believe we outperformed GHA when we did so.
We moved to GHA b/c nobody ever got fired ^W^W^W^W leadership thought eng running CI was not a good use of eng time. (Without much question into how much time was actually spent on it… which was pretty close to none. Self-hosted stuff has high initial cost for the setup … and then just kinda runs.)
Ironically, one of our self-hosted CI outages was caused by Azure — we have to get VMs from somewhere, and Azure … simply ran out. We had to swap to a different AZ to merely get compute.
The big upside to a self-hosted solution is that when stuff breaks, you can hold someone over the fire. (Above, that would be me, unfortunately.) With Github? Nobody really cares unless it is so big, and so severe, that they're more or less forced to, and even then, the response is usually lackluster.
Re: GitHub: Git operation failures
#256Reflecting on the last decade, with my career spanning big tech and startups, I've seen a common arch:
Small and scrappy startup -> taking on bigger customers for greater profits / ARR -> re-architecting for "enterprise" customers and resiliency / scale -> more idealism in engineering -> profit chasing -> product bloat -> good engineers leave -> replaced by other engineers -> failures expand.
This may be an acceptable lifecycle for individual companies as they each follow the destiny of chasing profits ultimately. Now picture it though for all the companies we've architected on top of (AWS, CloudFlare, GCP, etc.) Even within these larger organizations, they are comprised of multiple little businesses (eg: EC2 is its own business effectively - people wise, money wise)
Having worked at a $big_cloud_provider for 7 yrs, I saw this internally on a service level. What started as a foundational service, grew in scale, complexity, and architected for resiliency, slowly eroded its engineering culture to chase profits. Fundamental services becoming skeletons of their former selves, all while holding up the internet.
There isn't a singular cause here, and I can't say I know what's best, but it's concerning as the internet becomes more centralized into a handful of players.
tldr: how much of one's architecture and resiliency is built on the trust of "well (AWS|GCP|CloudFlare) is too big to fail" or "they must be doing things really well"? The various providers are not all that different from other tech companies on the inside. Politics, pressure, profit seeking.
Re: GitHub: Git operation failures
#257Earlier quoted context omitted.
Tbh, I personally don't trust a stopped escalator. Some of the videos of brake failures on them scared me off of ever going on them.
You've ruined something for me. My adult side is grateful but the rest of me is throwing a tantrum right now. I hope you're happy with what you've done.
I am genuinly sorry about that. And no, I am not happy about what I've done.
Re: GitHub: Git operation failures
#258What's the local workaround for this?
Git is distributed, it should be possible to put something between our servers and github which pulls from github when it's running and otherwise serves whatever it used to have. A cache of some sort. I've found the five year old https://github.com/jonasmalacofilho/git-cache-http-server which is the same sort of idea.
I've run a git instance on a local machine which I pull from, where a cron job fetches from upstream into it, which solved the problem of cloning llvm over a slow connection, so it's doable on a per-repo basis.
I'd like to replace it globally though because CI looks like "pull from loads of different git repos" and setting it up once per-repo seems dreadful. Once per github/gitlab would be a big step forward.
Re: GitHub: Git operation failures
#259> We are seeing failures for some git http operations and are investigating It's not just HTTPS, I can't push via SSH either. I'm not convinced it's just "some" operations either; every single one I've tried fails.
I'm convinced the people who write status pages are incapable of escaping the phrasing "Some users may be experiencing problems". Too much attempting to save face by PR types, instead of just being transparent with information (… which is what would actually save face…)
And that's if you get a status page update at all.
Re: GitHub: Git operation failures
#260Earlier quoted context omitted.
Microsoft is also convinced that its works are a net benefit for humanity, so I would take that with a grain of salt.
I think it would be pretty hard to argue against that point of view, at least thus far. If DOS/Windows hadn't become the dominant OS someone would have, and a whole generation of engineers cut their teeth on their parents' windows PCs.
And how does it follow that microsoft is the good guy in a future where we did it with some other operating system? You could argue that their system was so terrible that its displacement of other options harmed us all with the same level of evidence.