Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

GitHub: Git operation failures

githubstatus.com

Post reply on HN
Print view View original

261–270 of 335 posts

Re: GitHub: Git operation failures

#261

Post by projproj »

angrydev wrote: post #143
projproj wrote: post #50

Obviously just speculation, but maybe don't let AI write your code... Microsoft CEO says up to 30% of the company’s code was written by AI https://techcrunch.com/2025/04/29/microsoft-ceo-says-up-to-3...

What a ridiculous comment, as if these outages didn't happen before LLMs became more commonplace.

I admit it was a bit ridiculous. However, if Microsoft is going to brag about how much AI code they are using but not also brag about how good the code is, then we are left to speculate. The two outages in two weeks are _possible_ data points and all we have to go on unless they start providing data.

Re: GitHub: Git operation failures

#262

Post by stinkbeetle »

HardwareLust wrote: post #241
aeldidi wrote: post #238

I'm becoming concerned with the rate at which major software systems seem to be failing as of late. For context, last year I only logged four outages that actually disrupted my work; this quarter alone I'm already on my fourth, all within the past few weeks. This is, of course, just an anecdote and not evidence of any wider trend (not to mention that I might not have even logged everything last year), but it was enou…

It's money, of course. No one wants to pay for resilience/redundancy. I've launched over a dozen projects going back to 2008, clients simply refuse to pay for it, and you can't force them. They'd rather pinch their pennies, roll the dice and pray.

> It's money, of course.

100%

> No one wants to pay for resilience/redundancy. I've launched over a dozen projects going back to 2008, clients simply refuse to pay for it, and you can't force them. They'd rather pinch their pennies, roll the dice and pray.

Well, fly by night outfits will do that. Bigger operations like GitHub will try to do the math on what an outage costs vs what better reliability costs, and optimize accordingly.

Look at a big bank or a big corporation's accounting systems, they'll pay millions just for the hot standby mainframes or minicomputers that, for most of them, would never be required.

Re: GitHub: Git operation failures

#263

Post by Esophagus4 »

captainkrtek wrote: post #256

Reflecting on the last decade, with my career spanning big tech and startups, I've seen a common arch: Small and scrappy startup -> taking on bigger customers for greater profits / ARR -> re-architecting for "enterprise" customers and resiliency / scale -> more idealism in engineering -> profit chasing -> product bloat -> good engineers leave -> replaced by other engineers -> failures expand. This may be an acceptabl…

Well said. I definitely agree (you’re absolutely right!) that the product will get worse through that re-architecting for enterprise transition.

But the small product also would not be able to handle any real amount of growth as it was, because it was a mess of tech debt and security issues and manual one-off processes and fragile spaghetti code that only Jeff knows because he wrote it in a weekend, and now he’s gone.

So by definition, if a service is large enough to serve a zillion people, it is probably big and bloated and complex.

I’m not disagreeing with you, I liked your comment and I’m just rambling. I have worked with several startups and was surprised at how poorly their tech scaled (and how riddled with security issues they were) as we got into it.

Nothing will shine a flashlight on all the stress cracks of a system like large-scale growth on the web.

Re: GitHub: Git operation failures

#264

Post by sshine »

wavemode wrote: post #242
tyre wrote: post #234

Earlier quoted context omitted.

My reason for centralized cloud solutions is also uptime. Multi-AZ RDS is 100% higher availability than me managing something.

Well, just a few weeks ago we weren't able to connect to RDS for several hours. That's way more downtime than we ever had at the company I worked for 10 years ago, where the DB was just running on a computer in the basement. Anecdotal, but ¯\_(ツ)_/¯

An anecdote that repeats.

Most software doesn’t need to be distributed. But it’s the growth paradigm where we build everything on principles that can scale to world-wide low-latency accessibility.

A UNIX pipe gets replaced with a $1200/mo. maximum IOPS RDS channel, bandwidth not included in price. Vendor lock-in guaranteed.

Re: GitHub: Git operation failures

#265

Post by Jenk »

stinkbeetle wrote: post #262
HardwareLust wrote: post #241

Earlier quoted context omitted.

It's money, of course. No one wants to pay for resilience/redundancy. I've launched over a dozen projects going back to 2008, clients simply refuse to pay for it, and you can't force them. They'd rather pinch their pennies, roll the dice and pray.

> It's money, of course. 100% > No one wants to pay for resilience/redundancy. I've launched over a dozen projects going back to 2008, clients simply refuse to pay for it, and you can't force them. They'd rather pinch their pennies, roll the dice and pray. Well, fly by night outfits will do that. Bigger operations like GitHub will try to do the math on what an outage costs vs what better reliability costs, and optimi…

I've worked at many big banks and corporations. They are all held together with the proverbial sticky tape, bubblegum, and hope.

They do have multiple layers of redundancies, and thus have the big budgets, but they won't be kept hot, or there will be some critical flaws that all of the engineers know about but they haven't been given permission/funding to fix, and are so badly managed by the firm, they dgaf either and secretly want the thing to burn.

There will be sustained periods of downtime if their primary system blips.

They will all still be dependent on some hyper-critical system that nobody really knows how it works, the last change was introduced in 1988 and it (probably) requires a terminal emulator to operate.

Re: GitHub: Git operation failures

#266

Post by captainkrtek »

Esophagus4 wrote: post #263
captainkrtek wrote: post #256

Reflecting on the last decade, with my career spanning big tech and startups, I've seen a common arch: Small and scrappy startup -> taking on bigger customers for greater profits / ARR -> re-architecting for "enterprise" customers and resiliency / scale -> more idealism in engineering -> profit chasing -> product bloat -> good engineers leave -> replaced by other engineers -> failures expand. This may be an acceptabl…

Well said. I definitely agree (you’re absolutely right!) that the product will get worse through that re-architecting for enterprise transition. But the small product also would not be able to handle any real amount of growth as it was, because it was a mess of tech debt and security issues and manual one-off processes and fragile spaghetti code that only Jeff knows because he wrote it in a weekend, and now he’s gone…

> So by definition, if a service is large enough to serve a zillion people, it is probably big and bloated and complex.

Totally agree with your take as well.

I think the unfortunate thing is that there can exist a "goldie locks zone" to this, where the service is capable of serving a zillion people AND is well architected. Unfortunately it can't seem to last forever.

I saw this in my career. More product SKUs were developed, new features/services defined by non-technical PMs, MBAs entered the chat, sales became the new focus over availability, and the engineering culture that made this possible eroded day by day.

The years I worked in this "goldie locks zone" I'd attribute to:

- strong technical leadership at the SVP+ level that strongly advocated for security, availability, then features (in that order).

- a strong operational culture. Incidents were exciting internally, post mortems shared at a company wide level, no matter how small.

- recognition for the engineers who chased ambulances and kept things running, beyond their normal job, this inspired others to follow in their footsteps.

Re: GitHub: Git operation failures

#268

Post by stinkbeetle »

Jenk wrote: post #265
stinkbeetle wrote: post #262

Earlier quoted context omitted.

> It's money, of course. 100% > No one wants to pay for resilience/redundancy. I've launched over a dozen projects going back to 2008, clients simply refuse to pay for it, and you can't force them. They'd rather pinch their pennies, roll the dice and pray. Well, fly by night outfits will do that. Bigger operations like GitHub will try to do the math on what an outage costs vs what better reliability costs, and optimi…

I've worked at many big banks and corporations. They are all held together with the proverbial sticky tape, bubblegum, and hope. They do have multiple layers of redundancies, and thus have the big budgets, but they won't be kept hot, or there will be some critical flaws that all of the engineers know about but they haven't been given permission/funding to fix, and are so badly managed by the firm, they dgaf either an…

I've worked on software used by these and have been called in to help support from time to time. One customer which is a top single digit public company by market cap (they may have been #1 at the time, a few years ago) had their SAP systems go down once every few days. This wasn't causing a real monetary problem for them because their hot standby took over.

They weren't using mainframes, just "big iron" servers, but each one would have been north of $5 million for the box alone, I guess on a 5ish year replacement schedule. Then there's all the networking, storage, licensing, support, and internal administration costs for it which would easily cost that much again.

Now people will say SAP systems are made entirely of dict tape and bubblegum. But it all worked. This system ran all their sales/purchasing sites and portals and was doing a million dollars every couple of minutes so that all paid for itself many times over during the course of that bug. Cold standby would not have cut it. Especially since these big systems take many minutes to boot and HANA takes even longer to load from storage.

Re: GitHub: Git operation failures

#269

Post by geoffbp »

_jab wrote: post #44

GitHub is pretty easily the most unreliable service I've used in the past five years. Is GitLab better in this regard? At this point my trust in GitHub is essentially zero - they don't deserve my money any longer.

Gitlab has regular issues (we use Saas) and the support isn’t great. They acknowledge problems, but the same ones happen again and again. It’s very hard to get anything on their roadmap etc.

Re: GitHub: Git operation failures

#270

Post by andrewstuart2 »

goda90 wrote: post #169
andrewstuart2 wrote: post #142

Earlier quoted context omitted.

I think it would be pretty hard to argue against that point of view, at least thus far. If DOS/Windows hadn't become the dominant OS someone would have, and a whole generation of engineers cut their teeth on their parents' windows PCs.

What if that alternate someone had been better than DOS/Windows and then engineers cut their teeth on that instead?

Then my comment may have been about a different OS. Or I might never have been born. Who knows?
Post reply on HN