Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

GitHub Outage Tracker: Is GitHub Cooked?

isgithubcooked.com

Post reply on HN
Print view View original

41–50 of 195 posts

Re: GitHub Outage Tracker: Is GitHub Cooked?

#41

Post by gen220 »

6LLvveMx2koXfwn wrote: post #35
JeremyHerrman wrote: post #21

> "GitHub has had 1125 incidents since February 2016, implying a monthly incident rate of 24" 1125 incidents / 126 months ≈ 8.9 incidents per month, not 24 still terrible, but why such an obvious error in the first sentence...

Not sure whether it has been updated since your comment, but the sentence now reads: GitHub has had 1125 incidents since March 2016. Over the last 3 months, they've averaged 24 incidents per month edit: although they also have 1.2 days of downtime (in a day) for their 'worst days' of downtime table, which suggests some auto number crunching is not working as expected.

Yes I tweaked it! The number and copy were mismatched and are no longer!

That worst day is likely an overlapping incidents accounting issue; I tried to account for overlapping incidents in another view but probably failed to port it over there.

Should be fixed soon!

Re: GitHub Outage Tracker: Is GitHub Cooked?

#42

Post by theideaofcoffee »

kashnote wrote: post #12

I think we need to have a little more sympathy for GitHub. You could justify the jabs when we could all blame any outage on the migration to Azure, but then they shared numbers around the scale they're dealing with now that everyone is constantly building and pushing with AI. I think it's commendable that they're not limiting access to the site or (intentionally) throttling newcomers. Yes, they need to get this figur…

Give me a break. Sympathy? For microsoft? That might have flown when github was like seven people, but they have nearly unlimited resources to make it better. They're just choosing not to. Let's talk contracts and money before we pull the sympathy card.

I used to be on-call in a high-traffic environment where single customers pushed more bits than entire nations. I chose the role. I didn't want people's sympathy, if anything, I wanted them to complain to management.

If it gets too bad they can quit. Maybe that would be for the best, just wear the thing down until it outright fails and no one wants to touch it. One less bullshit service sucking all of the oxygen out.

Re: GitHub Outage Tracker: Is GitHub Cooked?

#43

Post by niltecedu »

kashnote wrote: post #12

I think we need to have a little more sympathy for GitHub. You could justify the jabs when we could all blame any outage on the migration to Azure, but then they shared numbers around the scale they're dealing with now that everyone is constantly building and pushing with AI. I think it's commendable that they're not limiting access to the site or (intentionally) throttling newcomers. Yes, they need to get this figur…

idk man, we pay stupid amounts of money to microsoft, we are an enterprise customer, expecting better availablity compared to my laptop isnt really a high bar.

Re: GitHub Outage Tracker: Is GitHub Cooked?

#44

Post by jraph »

kashnote wrote: post #12

I think we need to have a little more sympathy for GitHub. You could justify the jabs when we could all blame any outage on the migration to Azure, but then they shared numbers around the scale they're dealing with now that everyone is constantly building and pushing with AI. I think it's commendable that they're not limiting access to the site or (intentionally) throttling newcomers. Yes, they need to get this figur…

They did that to themselves, and doubly so.

They decided they needed to capture the whole open source ecosystem by turning open source work into social networking... on a proprietary platform (because open source is great, especially when it's others' software). That was before they joined Microsoft.

And then Microsoft pushed AI everywhere, including on GitHub itself with copilot.

I would have liked if they had left the open source projects alone and didn't create that FOMO for not using them.

I have no sympathy.

Re: GitHub Outage Tracker: Is GitHub Cooked?

#45

Post by bamboozled »

kashnote wrote: post #12

I think we need to have a little more sympathy for GitHub. You could justify the jabs when we could all blame any outage on the migration to Azure, but then they shared numbers around the scale they're dealing with now that everyone is constantly building and pushing with AI. I think it's commendable that they're not limiting access to the site or (intentionally) throttling newcomers. Yes, they need to get this figur…

No, we don’t.

Re: GitHub Outage Tracker: Is GitHub Cooked?

#46

Post by Normal_gaussian »

nightpool wrote: post #24

Getting rid of Actions and Copilot and other secondary services almost halves Github's incident rate: https://i.imgur.com/XPcMIFr.png I'm a big fan of Github Actions and I think people are often a little too harsh on it, but it's clear that it's sad that it's come at such a high cost to the platform's stability

Actions aren't secondary to most paid GH users; and if they are down it usually means no deployments and no tests, which can often halt work.

Re: GitHub Outage Tracker: Is GitHub Cooked?

#47

Post by VCFundedGenYer »

kashnote wrote: post #12

I think we need to have a little more sympathy for GitHub. You could justify the jabs when we could all blame any outage on the migration to Azure, but then they shared numbers around the scale they're dealing with now that everyone is constantly building and pushing with AI. I think it's commendable that they're not limiting access to the site or (intentionally) throttling newcomers. Yes, they need to get this figur…

"Leave the big trillion dollar corporation alone" is not the right idea.

Re: GitHub Outage Tracker: Is GitHub Cooked?

#48

Post by graypegg »

JeremyHerrman wrote: post #21

> "GitHub has had 1125 incidents since February 2016, implying a monthly incident rate of 24" 1125 incidents / 126 months ≈ 8.9 incidents per month, not 24 still terrible, but why such an obvious error in the first sentence...

Ahhh, I think the author mixed up two values here. That value seems to actually be the average over the past 3 months.

    const incidents = e.detail.incidents;
    
    // ...snip...
    
    const now = new Date();
    const threeMonthsAgo = new Date(now);
    threeMonthsAgo.setMonth(threeMonthsAgo.getMonth() - 3);
    
    // ...snip...
    
    var currentFreq = recentIncidents.length / 3; //
#n-since is going to be either march or feburary. It'll change depending on your timezone because JS's Date object always shifts the date around to match the same instant but in the system's timezone.

#n-rate has nothing to do with the #n-since month, it's just the last trailing 3 months. And even then, it's sort of underbaked? It's moving the date back by 3 calendar months not taking into account differing numbers of days, so it'll under-report short months.

I wouldn't trust the stats here.

Edit: whoops, author updated the template while I was writing this! It now says "Over the last 3 months", though that's still calendar months.

Re: GitHub Outage Tracker: Is GitHub Cooked?

#49

Post by isityettime »

nightpool wrote: post #24

Getting rid of Actions and Copilot and other secondary services almost halves Github's incident rate: https://i.imgur.com/XPcMIFr.png I'm a big fan of Github Actions and I think people are often a little too harsh on it, but it's clear that it's sad that it's come at such a high cost to the platform's stability

GitHub Actions is really, really badly designed. The security model is fundamentally broken, the YAML hell is as bad as any, the log streaming lags like hell, they charge self-hosted runners for using their coordination plane, jobs queuing is really slow, and their software for actually running jobs is cursed and designed in a way that is practically hostile to self-hosting.

GitHub Actions is god-awful. Have you ever used any other CI tools?

Re: GitHub Outage Tracker: Is GitHub Cooked?

#50

Post by kstrauser »

jraph wrote: post #44
kashnote wrote: post #12

I think we need to have a little more sympathy for GitHub. You could justify the jabs when we could all blame any outage on the migration to Azure, but then they shared numbers around the scale they're dealing with now that everyone is constantly building and pushing with AI. I think it's commendable that they're not limiting access to the site or (intentionally) throttling newcomers. Yes, they need to get this figur…

They did that to themselves, and doubly so. They decided they needed to capture the whole open source ecosystem by turning open source work into social networking... on a proprietary platform (because open source is great, especially when it's others' software). That was before they joined Microsoft. And then Microsoft pushed AI everywhere, including on GitHub itself with copilot. I would have liked if they had left…

> And then Microsoft pushed AI everywhere, including on GitHub itself with copilot.

Ding ding ding, we have a winner. I like AI. I work for an AI company. Still, Microsoft aggressively pushed GitHub users toward Copilot. They don't get to do that and complain about increased volume from AI-generated changes.

No Copilot + reasonable operation: the way things were

Copilot + reasonable operation: Well done!

No Copilot + being overwhelmed by AI commits: Sympathy.

Copilot + being overwhelmed by AI commits: "Where did that petard come from that's hoisting us?"

Post reply on HN