Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

GitHub Outage Tracker: Is GitHub Cooked?

isgithubcooked.com

Post reply on HN
Print view View original

101–110 of 195 posts

Re: GitHub Outage Tracker: Is GitHub Cooked?

#101

Post by hansvm »

tyre wrote: post #58
gogobio wrote: post #23

Earlier quoted context omitted.

Sympathy? It's a Microsoft company that is being ran with a consistency of a startup in early seed rounds. Their downtime is abhorrent and unacceptable as far as enterprise goes. Their engineers look like absolute amateurs allowing for such low class work it results in their customers experiencing industry leading downtime.

This is uncharitable, rude, and pretty baseless. Unless you have a lot of direct, personal information about GitHub engineers, they’re dealing with a huge spike in traffic. Is their uptime acceptable? No. But personal attacks aren’t necessary or constructive.

The phrasing could have been better, but we've talked as a team about cancelling a $100k/yr contract nearly half a dozen times the last year, and the only thing keeping us from doing so is various "compliance" issues, and to a lesser degree the friction from physically moving. That's a tenuous moat, and if I were a GH PM I'd probably want to know that the current instability is somewhere near critical mass.

Re: GitHub Outage Tracker: Is GitHub Cooked?

#102

Post by [deleted] »

isityettime wrote: post #49
nightpool wrote: post #24

Getting rid of Actions and Copilot and other secondary services almost halves Github's incident rate: https://i.imgur.com/XPcMIFr.png I'm a big fan of Github Actions and I think people are often a little too harsh on it, but it's clear that it's sad that it's come at such a high cost to the platform's stability

GitHub Actions is really, really badly designed. The security model is fundamentally broken, the YAML hell is as bad as any, the log streaming lags like hell, they charge self-hosted runners for using their coordination plane, jobs queuing is really slow, and their software for actually running jobs is cursed and designed in a way that is practically hostile to self-hosting. GitHub Actions is god-awful. Have you ever…

[deleted]

Re: GitHub Outage Tracker: Is GitHub Cooked?

#103

Post by olyjohn »

majormajor wrote: post #81
joshuahedlund wrote: post #57

Given that the outages are caused by record traffic, so far they are only cooked in the Yogi Berra sense of “Nobody goes there anymore, it’s too crowded”

Random AI slop code pushers don't pay the bills. Traffic isn't revenue. I've started suggesting to my CTO that we move to something else because we do pay the Github bills and yet we are also affected by other people's activity . Avoiding that is not a hard engineering problem for an alternative. A world where a record amount of code is pushed to GH but a large amount of paying customers leave would be "cooked"

Yeah I don't feel bad for Microsoft, but I don't feel bad for people who keep paying them for it either. This has been going on forever now. Nothing will happen until people start leaving. I don't honestly understand why so much shit is on GitHub when there are so many alternatives, anyways. It's fucking git! Can't people do or try anything different, ever?

Re: GitHub Outage Tracker: Is GitHub Cooked?

#104

Post by chris_money202 »

cr125rider wrote: post #91
gogobio wrote: post #23

Earlier quoted context omitted.

Sympathy? It's a Microsoft company that is being ran with a consistency of a startup in early seed rounds. Their downtime is abhorrent and unacceptable as far as enterprise goes. Their engineers look like absolute amateurs allowing for such low class work it results in their customers experiencing industry leading downtime.

That seems a bit crass but the underlying sentiment stands. Microsoft has more money than God. 4T valuation. When you say you’re worth that much, no excuses. Figure it out.

So your proposal is for Microsoft to invest their free cash in helping support a bunch of AI coders maintain their pet projects? Sounds like you should be a CEO!

Re: GitHub Outage Tracker: Is GitHub Cooked?

#105

Post by chris_money202 »

rgbrenner wrote: post #71
kashnote wrote: post #12

I think we need to have a little more sympathy for GitHub. You could justify the jabs when we could all blame any outage on the migration to Azure, but then they shared numbers around the scale they're dealing with now that everyone is constantly building and pushing with AI. I think it's commendable that they're not limiting access to the site or (intentionally) throttling newcomers. Yes, they need to get this figur…

Microsoft made $134B in net profit last year, and is currently valued at nearly $4T. They have the resources to solve this if they wanted to.

Both Microsoft and GH issue is capacity which is being solved with huge capex in DC investment, you can't create a new DC overnight and its even harder in the current climate where everyone assumes the DC is for AI and not just normal workload compute like what GH needs...

Re: GitHub Outage Tracker: Is GitHub Cooked?

#107

Post by chris_money202 »

JDups wrote: post #27
kashnote wrote: post #12

I think we need to have a little more sympathy for GitHub. You could justify the jabs when we could all blame any outage on the migration to Azure, but then they shared numbers around the scale they're dealing with now that everyone is constantly building and pushing with AI. I think it's commendable that they're not limiting access to the site or (intentionally) throttling newcomers. Yes, they need to get this figur…

Constant building with AI is something that they (Microsoft) promote and are heavily invested in.

And then get penalized for in the market when they say we need to build more DCs and spend huge capex... which is what GH needs right now

Re: GitHub Outage Tracker: Is GitHub Cooked?

#108

Post by ksenzee »

vrosas wrote: post #90
logicchains wrote: post #85

Earlier quoted context omitted.

How low would uptime have to get before you'd consider holding the engineers personally responsible for it?

Agreed. At some point to have to stop hugging your ops team and start firing them.

No, you hire more of them, you give them what they need to do their jobs, and you actually listen to their recommendations. It's not even complicated. It just takes time and money, neither of which companies want to spend on reliability until their customers scream very loud en masse.

Re: GitHub Outage Tracker: Is GitHub Cooked?

#109

Post by xyst »

kashnote wrote: post #12

I think we need to have a little more sympathy for GitHub. You could justify the jabs when we could all blame any outage on the migration to Azure, but then they shared numbers around the scale they're dealing with now that everyone is constantly building and pushing with AI. I think it's commendable that they're not limiting access to the site or (intentionally) throttling newcomers. Yes, they need to get this figur…

"sympathy" for one of the worst corporations in history is some insane corp bootlicking logic.

This is a multi billion dollar corporation. History of robbing and stealing from others of their labor or IP. History of enshittifying once great services.

Re: GitHub Outage Tracker: Is GitHub Cooked?

#110

Post by anonymousab »

kashnote wrote: post #12

I think we need to have a little more sympathy for GitHub. You could justify the jabs when we could all blame any outage on the migration to Azure, but then they shared numbers around the scale they're dealing with now that everyone is constantly building and pushing with AI. I think it's commendable that they're not limiting access to the site or (intentionally) throttling newcomers. Yes, they need to get this figur…

> they shared numbers around the scale they're dealing with now that everyone is constantly building and pushing with AI.

That is an active choice they are making, over and instead of any reliability for the rest of their users.

It is a problem they are embracing, and actively encouraging, for themselves.

Post reply on HN