Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

GitHub Outage Tracker: Is GitHub Cooked?

isgithubcooked.com

Post reply on HN
Print view View original

61–70 of 195 posts

Re: GitHub Outage Tracker: Is GitHub Cooked?

#61

Post by californical »

zvmaz wrote: post #56
gogobio wrote: post #23

Earlier quoted context omitted.

Sympathy? It's a Microsoft company that is being ran with a consistency of a startup in early seed rounds. Their downtime is abhorrent and unacceptable as far as enterprise goes. Their engineers look like absolute amateurs allowing for such low class work it results in their customers experiencing industry leading downtime.

> Their engineers look like absolute amateurs allowing for such low class work it results in their customers experiencing industry leading downtime. I gather that you have intimate and deep knowledge on the teams and the problems they try to solve there.

If I own a restaurant and buy bread from a supplier — BreadHub.

And 99% of the bread that I get is good but 1% of the loaves, they forgot to add flour. Consistently, for years, they always have loaves missing a key ingredient that I still end up paying for.

I can be pretty sure that BreadHub have a pretty major internal issue, and should probably be questioning their competence, regardless of their “scale”, and without any knowledge of the “problems they’re solving”

Re: GitHub Outage Tracker: Is GitHub Cooked?

#62

Post by tyre »

sandeepkd wrote: post #53
kashnote wrote: post #12

I think we need to have a little more sympathy for GitHub. You could justify the jabs when we could all blame any outage on the migration to Azure, but then they shared numbers around the scale they're dealing with now that everyone is constantly building and pushing with AI. I think it's commendable that they're not limiting access to the site or (intentionally) throttling newcomers. Yes, they need to get this figur…

I think somewhere down the line you have an answer to problem itself. Its for a while that companies are not into the business of sympathies, we should probably not get into that side for the topic. 1. Github has enterprise users who paid for the service, their day job requires Github to be available and working 2. Github has generous free tier which is the one which is exploring a lot more with the AI generated code…

I agree that they should reserve capacity for enterprises and paying users.

But I think they’re in a tough spot. GitHub has historically been a huge supporter, proponent, and provider for open source projects. Engineers are difficult customers, to say the least, and the community would likely freak tf out of the segmented traffic.

The logical, pragmatic, and justifiable answer doesn’t always align with your market.

Re: GitHub Outage Tracker: Is GitHub Cooked?

#63

Post by bluerooibos »

kashnote wrote: post #12

I think we need to have a little more sympathy for GitHub. You could justify the jabs when we could all blame any outage on the migration to Azure, but then they shared numbers around the scale they're dealing with now that everyone is constantly building and pushing with AI. I think it's commendable that they're not limiting access to the site or (intentionally) throttling newcomers. Yes, they need to get this figur…

> have a little more sympathy for the billion dollar company

No.

Their leadership went all in on AI and in the last blog post essentially admitted some missing test coverage for a critical path. Time to learn lessons and fix your vibe coded shitslop and stop using "user graph go up" as some kind of excuse.

Re: GitHub Outage Tracker: Is GitHub Cooked?

#64

Post by tyre »

californical wrote: post #61
zvmaz wrote: post #56

Earlier quoted context omitted.

> Their engineers look like absolute amateurs allowing for such low class work it results in their customers experiencing industry leading downtime. I gather that you have intimate and deep knowledge on the teams and the problems they try to solve there.

If I own a restaurant and buy bread from a supplier — BreadHub. And 99% of the bread that I get is good but 1% of the loaves, they forgot to add flour. Consistently, for years, they always have loaves missing a key ingredient that I still end up paying for. I can be pretty sure that BreadHub have a pretty major internal issue, and should probably be questioning their competence, regardless of their “scale”, and witho…

I’m not saying users don’t have a right to be pissed or aren’t justified in looking at other options.

I’m saying that GH is operating at a huge scale with (probably) lots of technical debt and a forced migration to new infrastructure.

I would be (and am) highly critical of leadership. I’m not going to make strong assertions about ICs without knowing their context. I’ve worked at a company with a sterling reputation for engineering excellence where brilliant ICs were kneecapped by poor leadership.

I think a lot of us, at one point or another in our careers, have worked with potato leadership that can be short-sighted or political. It isn’t a comment on the engineers.

Re: GitHub Outage Tracker: Is GitHub Cooked?

#65

Post by alwa »

kashnote wrote: post #12

I think we need to have a little more sympathy for GitHub. You could justify the jabs when we could all blame any outage on the migration to Azure, but then they shared numbers around the scale they're dealing with now that everyone is constantly building and pushing with AI. I think it's commendable that they're not limiting access to the site or (intentionally) throttling newcomers. Yes, they need to get this figur…

For that matter, I admire their relative transparency about their incidents. I can think of other big players who will cheerfully show green statuses across the board while everyone can see that their pants are down...

Re: GitHub Outage Tracker: Is GitHub Cooked?

#66

Post by xyzzy_plugh »

Is GitHub Cooked? Decidedly yes.

Migrating to something else has been raised as a concern during every engagement with clients and prospective clients in the past year.

"Do you use GitHub?"

"Yes though we'd like to move to something else, we just don't know what yet."

As soon as the next big thing shows up, they're done. And they know it.

Re: GitHub Outage Tracker: Is GitHub Cooked?

#67

Post by reassess_blind »

IcyWindows wrote: post #59
sandeepkd wrote: post #53

Earlier quoted context omitted.

I think somewhere down the line you have an answer to problem itself. Its for a while that companies are not into the business of sympathies, we should probably not get into that side for the topic. 1. Github has enterprise users who paid for the service, their day job requires Github to be available and working 2. Github has generous free tier which is the one which is exploring a lot more with the AI generated code…

Do we have a breakdown on how much traffic is free users vs paid users?

I’d be surprised if paid traffic was more than 1%.

Re: GitHub Outage Tracker: Is GitHub Cooked?

#68

Post by tryauuum »

californical wrote: post #61
zvmaz wrote: post #56

Earlier quoted context omitted.

> Their engineers look like absolute amateurs allowing for such low class work it results in their customers experiencing industry leading downtime. I gather that you have intimate and deep knowledge on the teams and the problems they try to solve there.

If I own a restaurant and buy bread from a supplier — BreadHub. And 99% of the bread that I get is good but 1% of the loaves, they forgot to add flour. Consistently, for years, they always have loaves missing a key ingredient that I still end up paying for. I can be pretty sure that BreadHub have a pretty major internal issue, and should probably be questioning their competence, regardless of their “scale”, and witho…

Please don't do it. Now somebody will come and will try to improve your bread analogy. We will be discussing bread for eons

Re: GitHub Outage Tracker: Is GitHub Cooked?

#69

Post by kayfox »

californical wrote: post #61
zvmaz wrote: post #56

Earlier quoted context omitted.

> Their engineers look like absolute amateurs allowing for such low class work it results in their customers experiencing industry leading downtime. I gather that you have intimate and deep knowledge on the teams and the problems they try to solve there.

If I own a restaurant and buy bread from a supplier — BreadHub. And 99% of the bread that I get is good but 1% of the loaves, they forgot to add flour. Consistently, for years, they always have loaves missing a key ingredient that I still end up paying for. I can be pretty sure that BreadHub have a pretty major internal issue, and should probably be questioning their competence, regardless of their “scale”, and witho…

I see you haven't experienced USFoods and Sysco.

Re: GitHub Outage Tracker: Is GitHub Cooked?

#70

Post by Aeolun »

kashnote wrote: post #12

I think we need to have a little more sympathy for GitHub. You could justify the jabs when we could all blame any outage on the migration to Azure, but then they shared numbers around the scale they're dealing with now that everyone is constantly building and pushing with AI. I think it's commendable that they're not limiting access to the site or (intentionally) throttling newcomers. Yes, they need to get this figur…

Yeah, all those 1100 commits per second. Like, is this the standard for 'scale of traffic' that we're impressed by these days? And it's not like this snuck up on them. The graph shows a very consistent trend that gave them months and months to respond.
Post reply on HN