Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

GitHub: Git operation failures

githubstatus.com

Post reply on HN
Print view View original

231–240 of 335 posts

Re: GitHub: Git operation failures

#231

Post by keybored »

mandus wrote: post #87

Good thing git was designed as a decentralized revision control system, so you don’t really need GitHub. It’s just a nice convenience

I don’t use GitHub that much. I think the thing about “oh no you have centralized on GitHub” point is a bit exaggerated.[1] But generally, thinking beyond just pushing blobs to the Internet, “decentralization” as in software that lets you do everything that is Not Internet Related locally is just a great thing. So I can never understand people who scoff at Git being decentralized just because “um, actually you end up pushing to the same repository”.

It would be great to also have the continuous build and test and whatever else you “need” to keep the project going as local alternatives as well. Of course.

[1] Or maybe there is just that much downtime on GitHub now that it can’t be shrugged off

Re: GitHub: Git operation failures

#232

Post by input_sh »

ecshafer wrote: post #121
_jab wrote: post #44

GitHub is pretty easily the most unreliable service I've used in the past five years. Is GitLab better in this regard? At this point my trust in GitHub is essentially zero - they don't deserve my money any longer.

We self host gitlab, so its very stable. But Gitlab also kind of is enterprise software. It hits every feature checkbox, but they aren't well integrated, and they are kind of half way done. I don't think its as smooth of an experience as Github personally, or as feature rich. But Gitlab can self host your project repos, cicd, issues, wikis, etc. and it does it at least okay .

I would argue GitLab CI/CD is miles ahead of the dumpster fire that is GitHub Actions. Also the homepage is actually useful, unlike GitHub's.

Re: GitHub: Git operation failures

#233

Post by rvnx »

collingreen wrote: post #229
Akronymus wrote: post #226

Earlier quoted context omitted.

Tbh, I personally don't trust a stopped escalator. Some of the videos of brake failures on them scared me off of ever going on them.

You've ruined something for me. My adult side is grateful but the rest of me is throwing a tantrum right now. I hope you're happy with what you've done.

I read a book about elevators accidents; don't.

Re: GitHub: Git operation failures

#234

Post by tyre »

wavemode wrote: post #217
IshKebab wrote: post #201

Earlier quoted context omitted.

Do you think you'd get better uptime with your own solution? I doubt it. It would just be at a different time.

Uptime is much, much easier at low scale than at high scale. The reason for buying centralized cloud solutions is not uptime, it's to safe the headache of developing and maintaining the thing.

My reason for centralized cloud solutions is also uptime.

Multi-AZ RDS is 100% higher availability than me managing something.

Re: GitHub: Git operation failures

#235

Post by prescriptivist »

IshKebab wrote: post #201
jimbokun wrote: post #170

Earlier quoted context omitted.

As long as you didn't go all in on GitHub Actions. Like my company has.

Do you think you'd get better uptime with your own solution? I doubt it. It would just be at a different time.

It's fairly straightforward to build resilient, affordable and scalable pipelines with DAG orchestrators like tekton running in kubernetes. Tekton in particular has the benefit of being low level enough that it can just be plugged into the CI tool above it (jenkins, argo, github actions, whatever) and is relatively portable.

Re: GitHub: Git operation failures

#236

Post by codethief »

Lammy wrote: post #151
wnevets wrote: post #99

Earlier quoted context omitted.

> A. Are these major issues with cloud/SaaS tools becoming more common, or is it just that they get a lot more coverage now? It seems like we see major issues across AWS, GCP, Azure, Github, etc. at least monthly now and I don't remember that being the case in the past. FWIW Microsoft is convinced moving Github to Azure will fix these outages

Everything old is new again. https://www.zdnet.com/article/ms-moving-hotmail-to-win2000-s... https://jimbojones.livejournal.com/23143.html

From the second link:

> In 2002, the amusement continued when a network security outfit discovered an internal document server wide open to the public internet in Microsoft's supposedly "private" network, and found, among other things, a whitepaper[0] written by the hotmail migration team explaining why unix is superior to windows.

Hahaha, that whitepaper is pure gold!

[0]: https://web.archive.org/web/20040401182755/http://www.securi...

Re: GitHub: Git operation failures

#237

Post by sznio »

consumer451 wrote: post #109

We live in a house of cards. I hope that eventually people in power realize this. However, their incentive structures do not seem to be a forcing function for that eventuality. I have been thinking about this a lot lately. What would be a tweak that might improve this situation?

Not exactly for this situation, but I've been thinking about distributed caching of web content.

Even if a website is down, someone somewhere most likely has it cached. Why can't I read it from their cache? If I'm trying to reach a static image file, why do I have to get it from the source?

I guess I want torrent DHT for the web.

Re: GitHub: Git operation failures

#238

Post by aeldidi »

I'm becoming concerned with the rate at which major software systems seem to be failing as of late. For context, last year I only logged four outages that actually disrupted my work; this quarter alone I'm already on my fourth, all within the past few weeks. This is, of course, just an anecdote and not evidence of any wider trend (not to mention that I might not have even logged everything last year), but it was enough to nudge me into writing this today (helped by the fact that I suddenly had some downtime). Keep in mind, this isn't necessarily specific to this outage, just something that's been on my mind enough to warrant writing about it.

It feels like resiliency is becoming a bit of a lost art in networked software. I've spent a good chunk of this year chasing down intermittent failures at work, and I really underestimated how much work goes into shrinking the "blast radius", so to speak, of any bug or outage. Even though we mostly run a monolith, we still depend on a bunch of external pieces like daemons, databases, Redis, S3, monitoring, and third-party integrations, and we generally assume that these things are present and working in most places, which wasn't always the case. My response was to better document the failure conditions, and once I did, realize that there was many more than we initially thought. Since then we've done things like: move some things to a VPS instead of cloud services, automate deployment more than we already had, greatly improve the test suite and docs to include these newly considered failure conditions, and generally cut down on moving parts. It was a ton of effort, but the payoff has finally shown up: our records show fewer surprises which means fewer distractions and a much calmer system overall. Without that unglamorous work, things would've only grown more fragile as complexity crept in. And I worry that, more broadly, we're slowly un-learning how to build systems that stay up even when the inevitable bug or failure shows up.

For completeness, here are the outages that prompted this: the AWS us-east-1 outage in October (took down the Lightspeed R series API), the Azure Front Door outage (prevented Playwright from downloading browsers for tests), today’s Cloudflare outage (took down Lightspeed’s website, which some of our clients rely on), and the Github outage affecting basically everyone who uses it as their git host.

Re: GitHub: Git operation failures

#239

Post by dlenski »

SteveNuts wrote: post #67

I have a serious question, not trying to start a flame war. A. Are these major issues with cloud/SaaS tools becoming more common, or is it just that they get a lot more coverage now? It seems like we see major issues across AWS, GCP, Azure, Github, etc. at least monthly now and I don't remember that being the case in the past. B. If it's becoming more common, what are the reasons? I can think of a few, but I don't kn…

> Are these major issues with cloud/SaaS tools becoming more common, or is it just that they get a lot more coverage now?

I think that "more coverage" is part of it, but also "more centralization." More and more of the web is centralized around a tiny number of cloud providers, because it's just extremely time-intensive and cost-prohibitive for all but the largest and most specialized companies to run their own datacenters and servers.

Three specific examples: Netflix and Dropbox do run their own datacenters and servers; Strava runs on AWS.

> If it's becoming more common, what are the reasons? I can think of a few, but I don't know the answer, so if anyone in-the-know has insight I'd appreciate it.

I worked at AWS from 2020-2024, and saw several of these outages so I guess I'm "in the know."

My somewhat-cynical take is that a lot of these services have grown enormously in complexity, far outstripping the ability of their staff to understand them or maintain them:

- The OG developers of most of these cloud services have moved on. Knowledge transfer within AWS is generally very poor, because it's not incentivized, and has gotten worse due to remote work and geographic dispersion of service teams.

- Managers at AWS are heavily incentivized to develop "new features" and not to improve the reliability, or even security, of their existing offerings. (I discovered numerous security vulnerabilities in the very-well-known service that I worked for, and was regularly punished-rather-than-rewarded for trying to get attention and resources on this. It was a big part of what drove me to leave Amazon. I'm still sitting on a big pile of zero-day vulnerabilities in ______ and ______.)

- Cloud services in most of the world are basically a 3-way oligopoly between AWS, Microsoft/Azure, and Google. The costs of switching from one provider to another are often ENORMOUS due to a zillion fiddly little differences and behavior quirks ("bugs"). It's not apparent to laypeople — or even to me — that any of these providers are much more or less reliable than the others.

Re: GitHub: Git operation failures

#240

Post by ambicapter »

tingletech wrote: post #219
SteveNuts wrote: post #67

I have a serious question, not trying to start a flame war. A. Are these major issues with cloud/SaaS tools becoming more common, or is it just that they get a lot more coverage now? It seems like we see major issues across AWS, GCP, Azure, Github, etc. at least monthly now and I don't remember that being the case in the past. B. If it's becoming more common, what are the reasons? I can think of a few, but I don't kn…

Years ago on hackernews I saw a link about probability describing a statistical technique that one could use to answer a question about if a specific type of event was becoming more common or not. Maybe related to the birthday paradox? The gist that I remember is that sometimes a rare event will seem to be happening more often, when in reality there is some cognitive bias that makes it non-intuitive to make that deci…

If the events are independent, you could use a binomial distribution. Not sure if you can consider these kinds of events to be independent, though.
Post reply on HN