Live data from Hacker News

GitHub Outage Tracker: Is GitHub Cooked?

isgithubcooked.com

181–190 of 195 posts

Re: GitHub Outage Tracker: Is GitHub Cooked?

#181
post #62

Earlier quoted context omitted.

I agree that they should reserve capacity for enterprises and paying users. But I think they’re in a tough spot. GitHub has historically been a huge supporter, proponent, and provider for open source projects. Engineers are difficult customers, to say the least, and the community would likely freak tf out of the segmented traffic. The logical, pragmatic, and justifiable answer doesn’t always align with your market.

Not sure if I would agree to that completely. Free tier is always ripe for abuse so in the beginning for a long time, Github has this requirement that you can only have a public repository if you are a non-paying user. Somewhere down the road when machine learning picked up (everyone was on the band wagon of hoarding the data, datalakes, hadoop, flink were at the top) they relaxed that condition allowing one to have…

Tbh, timing wise it appeared to me that GH started allowing private repos when one of the competitors started to allow it for free. I forget the name of the back then still competitor.. it worked on me, I retuned to GH back then..

Re: GitHub Outage Tracker: Is GitHub Cooked?

#182

Earlier quoted context omitted.

what's normal working hours? very US centric comment IMO. Europe, India, China, Latam etc. don't fall into your 9-5 EST normal working hour bucket.

Im fairly certain that SWEs only exist on the US west coast

They would be in PST then, no?

Re: GitHub Outage Tracker: Is GitHub Cooked?

#183
post #180
post #52

Near the end of the 8.5 years that I worked at GitHub as an enterprise support engineer, I asked in an all hands if a "GitHub Classic" product had been considered. Much like World of Warcraft Classic, I imagined it would be a rewrite focused on matching the simpler feature set of the past. I was basically given the same response that blizzard gave that question; "you think you want that but you don't".

Fun idea, I mean, I use old.reddit.com, would probably use classic.github.com too... Imagine YouTube Classic...

nice old.reddit.com, but I don’t think people want something as far back as 2015, they just want their forge to stay boring and fast

Re: GitHub Outage Tracker: Is GitHub Cooked?

#184
post #180
post #52

Near the end of the 8.5 years that I worked at GitHub as an enterprise support engineer, I asked in an all hands if a "GitHub Classic" product had been considered. Much like World of Warcraft Classic, I imagined it would be a rewrite focused on matching the simpler feature set of the past. I was basically given the same response that blizzard gave that question; "you think you want that but you don't".

Fun idea, I mean, I use old.reddit.com, would probably use classic.github.com too... Imagine YouTube Classic...

Frankly, even if it was limited only to videos contemporary to whichever point we decided is "classic", I'd still be in.

Re: GitHub Outage Tracker: Is GitHub Cooked?

#186
post #55

Earlier quoted context omitted.

We have the expertise. We don’t have the interest in fighting our internal “security” teams to keep running VMs.

Potato potato. Expertise isn't just "Do we have a sysadmin?", it's also "Does the org enable them to get their job done?".

You are missing the compliance. Enterprise is mostly compliance, making job done is just second

Re: GitHub Outage Tracker: Is GitHub Cooked?

#187

Given that the outages are caused by record traffic, so far they are only cooked in the Yogi Berra sense of “Nobody goes there anymore, it’s too crowded”

Random AI slop code pushers don't pay the bills. Traffic isn't revenue. I've started suggesting to my CTO that we move to something else because we do pay the Github bills and yet we are also affected by other people's activity . Avoiding that is not a hard engineering problem for an alternative. A world where a record amount of code is pushed to GH but a large amount of paying customers leave would be "cooked"

It is quite possible for github load balancers to route paying customers to different queues/infrastructure. I think much of the bot activity is paying customers, they just don't have per-client QoS because that would be hard. And they don't have separate auth for bots so they can't queue them separately.

Re: GitHub Outage Tracker: Is GitHub Cooked?

#188
post #13

If backend GitHub services are anything like GHES then I’m surprised it even managed to scale this much.

I think its nothing like the Enterprise version, which is why they hate having to maintain it, same with Atlassian and Azure/Entra AD. Once service to rule them all, and in the cloud bind them.

For security reasons, I would love a low feature, carefully engineered, offline github-like system. Something tuned for agentic behaviours, with an understanding of hierarchical agent identities.

Re: GitHub Outage Tracker: Is GitHub Cooked?

#189
post #180
post #52

Near the end of the 8.5 years that I worked at GitHub as an enterprise support engineer, I asked in an all hands if a "GitHub Classic" product had been considered. Much like World of Warcraft Classic, I imagined it would be a rewrite focused on matching the simpler feature set of the past. I was basically given the same response that blizzard gave that question; "you think you want that but you don't".

Fun idea, I mean, I use old.reddit.com, would probably use classic.github.com too... Imagine YouTube Classic...

Maybe off topic, but I'm very sad old.reddit.com is now forcing login. Have pretty much stopped using reddit since they did that.

Re: GitHub Outage Tracker: Is GitHub Cooked?

#190

Getting rid of Actions and Copilot and other secondary services almost halves Github's incident rate: https://i.imgur.com/XPcMIFr.png I'm a big fan of Github Actions and I think people are often a little too harsh on it, but it's clear that it's sad that it's come at such a high cost to the platform's stability

GitHub Actions is really, really badly designed. The security model is fundamentally broken, the YAML hell is as bad as any, the log streaming lags like hell, they charge self-hosted runners for using their coordination plane, jobs queuing is really slow, and their software for actually running jobs is cursed and designed in a way that is practically hostile to self-hosting. GitHub Actions is god-awful. Have you ever…

> they charge self-hosted runners for using their coordination plane

They don't. Self-hosted runners are still free. They backed down from their plan to charge after receiving widespread backlash.

Post reply on HN