Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

GitHub: Git operation failures

githubstatus.com

Post reply on HN
Print view View original

271–280 of 335 posts

Re: GitHub: Git operation failures

#271

Post by andrewstuart2 »

bovermyer wrote: post #168
andrewstuart2 wrote: post #142

Earlier quoted context omitted.

I think it would be pretty hard to argue against that point of view, at least thus far. If DOS/Windows hadn't become the dominant OS someone would have, and a whole generation of engineers cut their teeth on their parents' windows PCs.

I'm not convinced of your first point. Just because something seems difficult to avoid given the current context does not mean it was the only path available. Your second point is a little disingenuous. Yes, Microsoft and Windows have been wildly successful from a cultural adoption standpoint. But that's not the point I was trying to argue.

My first comment is simply pointing out that there's always a #1 in anything you can rank. Windows happened to be what won. And I learned how to use a computer on Windows. Do I use it now? No. But I learned on it as did most people whose parents wanted a computer.

Re: GitHub: Git operation failures

#272

Post by andrewstuart2 »

cdaringe wrote: post #171
andrewstuart2 wrote: post #142

Earlier quoted context omitted.

I think it would be pretty hard to argue against that point of view, at least thus far. If DOS/Windows hadn't become the dominant OS someone would have, and a whole generation of engineers cut their teeth on their parents' windows PCs.

There are some pretty zany alternative realities in the Multiverses I’ve visited. Xerox Parc never went under and developed computing as a much more accessible commodity. Another, Bell labs invented a whole category of analog computers that’s supplanted our universe’s digital computing era. There’s one where IBM goes directly to super computers in the 80s. While undoubtedly Microsoft did deliver for many of us, I am…

Yeah, I'm absolutely not saying it was the only path. It's just the path that happened. If not MS maybe it would have been Unix and something else. Either way most everyone today uses UX based on Xerox Parc's which was generously borrowed by, at this point, pretty much everyone.

Re: GitHub: Git operation failures

#274

Post by macintux »

lopatin wrote: post #248
HardwareLust wrote: post #241

Earlier quoted context omitted.

It's money, of course. No one wants to pay for resilience/redundancy. I've launched over a dozen projects going back to 2008, clients simply refuse to pay for it, and you can't force them. They'd rather pinch their pennies, roll the dice and pray.

I agree that it's all money. That's why it's always DNS right? > No one wants to pay for resilience/redundancy These companies do take it seriously, on the software side, but when it comes to configurations, what are you going to do: Either play it by ear, or literally double your cloud costs for a true, real prod-parallel to mitigate that risk. It looks like even the most critical and prestigious companies in the wo…

> Either play it by ear, or literally double your cloud costs for a true, real prod-parallel to mitigate that risk.

There's also the problem that doubling your cloud footprint to reduce the risk of a single point of failure introduces new risks: more configuration to break, new modes of failure when both infrastructures are accidentally live and processing traffic, etc.

Back when companies typically ran their own datacenters (or otherwise heavily relied on physical devices), I was very skeptical about redundant switches, fearing the redundant hardware would cause more problems than it solved.

Re: GitHub: Git operation failures

#275

Post by macintux »

Aperocky wrote: post #228
tcoff91 wrote: post #207

Earlier quoted context omitted.

Compared to 2025 github yeah I do think most self-hosted CI systems would be more available. Github goes down weekly lately.

Aren't they halting all work to migrate to azure? Does not sound like an easy thing to do and feels quite easy to cause unexpected problems.

I recall the Hotmail acquisition and the failed attempts to migrate the service to Windows servers.

Re: GitHub: Git operation failures

#276

Post by porridgeraisin »

ForHackernews wrote: post #249
HardwareLust wrote: post #241

Earlier quoted context omitted.

It's money, of course. No one wants to pay for resilience/redundancy. I've launched over a dozen projects going back to 2008, clients simply refuse to pay for it, and you can't force them. They'd rather pinch their pennies, roll the dice and pray.

Why should they? Honestly most of what we do simply does not matter that much. 99.9% uptime is fine in 99.999% of cases.

This is true. But unfortunately the exact same process is used even for critical stuff (the crowdstrike thing for example). Maybe there needs to be a separate swe process for those things as well, just like there is for aviation. This means not using the same dev tooling, which is a lot of effort.

Re: GitHub: Git operation failures

#277

Post by zdragnar »

suddenlybananas wrote: post #243
aeldidi wrote: post #238

I'm becoming concerned with the rate at which major software systems seem to be failing as of late. For context, last year I only logged four outages that actually disrupted my work; this quarter alone I'm already on my fourth, all within the past few weeks. This is, of course, just an anecdote and not evidence of any wider trend (not to mention that I might not have even logged everything last year), but it was enou…

To be deliberately provocative, LLMs are being more and more widely used.

Word on the street is github was already a giant mess before the rise of LLMs, and it has not improved with the move to MS.

Re: GitHub: Git operation failures

#278

Post by consumer451 »

sznio wrote: post #237
consumer451 wrote: post #109

We live in a house of cards. I hope that eventually people in power realize this. However, their incentive structures do not seem to be a forcing function for that eventuality. I have been thinking about this a lot lately. What would be a tweak that might improve this situation?

Not exactly for this situation, but I've been thinking about distributed caching of web content. Even if a website is down, someone somewhere most likely has it cached. Why can't I read it from their cache? If I'm trying to reach a static image file, why do I have to get it from the source? I guess I want torrent DHT for the web.

That is genuinely interesting. But, let's put all "this nerd talk" into terms that someone in the average C-suite could understand.

How can C-suite stock RSU/comp/etc be tweaked to make them give a crap about this, or security?

---

Decades ago, I was a teenager and I realized that going to fancy hotel bars was really interesting. I looked old enough, and I was dressed well. This was in Seattle. I once overheard a low-level cellular company exec/engineer complain about how he had to climb a tower, and check the radiation levels (yes non-ionizing). But this was a low level exec, who had to take responsibility.

He joked about how while checking a building on cap hill, he waved his wand above his head, and when he heard the beeps... he noped tf out. He said that it sucked that he had to do that, and sign-off.

That is actually cool, and real engineering/responsibility at the executive level.

Can we please get more of that type of thing?

Re: GitHub: Git operation failures

#279

Post by roxolotl »

aeldidi wrote: post #238

I'm becoming concerned with the rate at which major software systems seem to be failing as of late. For context, last year I only logged four outages that actually disrupted my work; this quarter alone I'm already on my fourth, all within the past few weeks. This is, of course, just an anecdote and not evidence of any wider trend (not to mention that I might not have even logged everything last year), but it was enou…

To agree with the comments it seems likely it's money which has begun to result in a slow "un-learning how to build systems that stay up even when the inevitable bug or failure shows up."

Re: GitHub: Git operation failures

#280

Post by grepfru_it »

There was a comment on another GitHub thread that I replied to. I got a response saying it’s absurd how unreliable Gh is when people depend on it for CI/CD. And I think this is the problem. At GitHub the developers think it’s only a problem because their ci/cd is failing. Oh no, we broke GitHub actions, the actions runners team is going to be mad at us! Instead of, oh no, we broke GitHub actions, half the world is down!

That larger view held only by a small sliver of employees is likely why reliability is not a concern. That leads to the every team for themselves mentality. “It’s not our problem, and we won’t make it our problem so we don’t get dinged at review time” (ok that is Microsoft attitude leaking)

Then there’s their entrenched status. Real talk, no one is leaving GitHub. So customers will suck it up and live with it while angry employees grumble on an online forum. I saw this same attitude in major companies like Verio and Verisign in the early 2000s. “Yeah we’re down but who else are you going to go to? Have a 20% discount since you complained. We will only be 1% less profitable this quarter due to it” The kang and kodos argument personified.

These views are my own and not related to my employer or anyone associated with me.

Post reply on HN