Live data from Hacker News

GitHub Outage Tracker: Is GitHub Cooked?

isgithubcooked.com

51–60 of 195 posts

Re: GitHub Outage Tracker: Is GitHub Cooked?

#51

I think we need to have a little more sympathy for GitHub. You could justify the jabs when we could all blame any outage on the migration to Azure, but then they shared numbers around the scale they're dealing with now that everyone is constantly building and pushing with AI. I think it's commendable that they're not limiting access to the site or (intentionally) throttling newcomers. Yes, they need to get this figur…

It’s commendable that they let people give them training data for free?

Re: GitHub Outage Tracker: Is GitHub Cooked?

#52
Near the end of the 8.5 years that I worked at GitHub as an enterprise support engineer, I asked in an all hands if a "GitHub Classic" product had been considered. Much like World of Warcraft Classic, I imagined it would be a rewrite focused on matching the simpler feature set of the past.

I was basically given the same response that blizzard gave that question; "you think you want that but you don't".

Re: GitHub Outage Tracker: Is GitHub Cooked?

#53

I think we need to have a little more sympathy for GitHub. You could justify the jabs when we could all blame any outage on the migration to Azure, but then they shared numbers around the scale they're dealing with now that everyone is constantly building and pushing with AI. I think it's commendable that they're not limiting access to the site or (intentionally) throttling newcomers. Yes, they need to get this figur…

I think somewhere down the line you have an answer to problem itself. Its for a while that companies are not into the business of sympathies, we should probably not get into that side for the topic.

1. Github has enterprise users who paid for the service, their day job requires Github to be available and working

2. Github has generous free tier which is the one which is exploring a lot more with the AI generated code.

It is a complexity in itself but the traffic should have been separated, the free users should not be allowed to bring down Github for enterprise customers (Just to clarify, I am free user myself). And if they do not have capacity it would have been perfectly fine to push back or throttle new users/repositories.

Re: GitHub Outage Tracker: Is GitHub Cooked?

#54

I think we need to have a little more sympathy for GitHub. You could justify the jabs when we could all blame any outage on the migration to Azure, but then they shared numbers around the scale they're dealing with now that everyone is constantly building and pushing with AI. I think it's commendable that they're not limiting access to the site or (intentionally) throttling newcomers. Yes, they need to get this figur…

Our company pays for GitHub. We're paying for a broken product that stops our work. I don't have sympathy for the companies whom I pay for a product and give me broken shit in return. This is entirely preventable and their own fault.

A restaurant makes pizzas. They suddenly get 100x more popular. They can't make 100x more pizzas. But they are still taking orders from 100x more people. Not only are they not getting enough pizzas delivered that they took orders for, but in their rush to make and deliver more pizzas, they set the kitchen on fire, which makes an even longer wait for pizzas.

When the pizza you ordered doesn't get delivered, do you have sympathy for the restaurant? Or do you tell them to stop taking orders they can't fill and try not to set the kitchen on fire?

Now consider the pizza restaurant has 21 billion dollars in cash, is taking your money, and not giving you pizza.

Re: GitHub Outage Tracker: Is GitHub Cooked?

#55
post #17

Earlier quoted context omitted.

It's a shame that the GitHub org that we use at my job that we pay a lot of money for gets affected the same way my personal nonsense does. I don't know the architecture or any of that, but I feel like there could be (and it's not like they would've really known this until the last year or two with the massive spike) separate infrastructure for paid users/orgs vs free the same way they make the distinction with enter…

Our GitlabCE instance has been sitting in the racks for nearly two years with almost 100% uptime running on 10 year old xeons that have long since paid for themselves. Of course it is not free of all management but for our use case it is working. There are hiccups with the CI runners from time time but nothing major and we have another machine in another rack that serves as a backup which can be brought up ~ I know c…

We have the expertise. We don’t have the interest in fighting our internal “security” teams to keep running VMs.

Re: GitHub Outage Tracker: Is GitHub Cooked?

#56
post #23

I think we need to have a little more sympathy for GitHub. You could justify the jabs when we could all blame any outage on the migration to Azure, but then they shared numbers around the scale they're dealing with now that everyone is constantly building and pushing with AI. I think it's commendable that they're not limiting access to the site or (intentionally) throttling newcomers. Yes, they need to get this figur…

Sympathy? It's a Microsoft company that is being ran with a consistency of a startup in early seed rounds. Their downtime is abhorrent and unacceptable as far as enterprise goes. Their engineers look like absolute amateurs allowing for such low class work it results in their customers experiencing industry leading downtime.

> Their engineers look like absolute amateurs allowing for such low class work it results in their customers experiencing industry leading downtime.

I gather that you have intimate and deep knowledge on the teams and the problems they try to solve there.

Re: GitHub Outage Tracker: Is GitHub Cooked?

#58
post #23

I think we need to have a little more sympathy for GitHub. You could justify the jabs when we could all blame any outage on the migration to Azure, but then they shared numbers around the scale they're dealing with now that everyone is constantly building and pushing with AI. I think it's commendable that they're not limiting access to the site or (intentionally) throttling newcomers. Yes, they need to get this figur…

Sympathy? It's a Microsoft company that is being ran with a consistency of a startup in early seed rounds. Their downtime is abhorrent and unacceptable as far as enterprise goes. Their engineers look like absolute amateurs allowing for such low class work it results in their customers experiencing industry leading downtime.

This is uncharitable, rude, and pretty baseless. Unless you have a lot of direct, personal information about GitHub engineers, they’re dealing with a huge spike in traffic.

Is their uptime acceptable? No. But personal attacks aren’t necessary or constructive.

Re: GitHub Outage Tracker: Is GitHub Cooked?

#59

I think we need to have a little more sympathy for GitHub. You could justify the jabs when we could all blame any outage on the migration to Azure, but then they shared numbers around the scale they're dealing with now that everyone is constantly building and pushing with AI. I think it's commendable that they're not limiting access to the site or (intentionally) throttling newcomers. Yes, they need to get this figur…

I think somewhere down the line you have an answer to problem itself. Its for a while that companies are not into the business of sympathies, we should probably not get into that side for the topic. 1. Github has enterprise users who paid for the service, their day job requires Github to be available and working 2. Github has generous free tier which is the one which is exploring a lot more with the AI generated code…

Do we have a breakdown on how much traffic is free users vs paid users?

Re: GitHub Outage Tracker: Is GitHub Cooked?

#60
I feel for the team having to deal with these problems at github today. Its also beyond me how those in charge for years wouldn't have seen this coming.

Github was bought by Microsoft, whether they wanted to acknowledge that internally or not. Microsoft went deep on LLMs, and specifically on LLMs for coding use cases. They must have recognized that LLM generated code and PRs would effectively DDoS GitHub.

I can only assume they simply didn't care, likely driven by greed.

Post reply on HN