Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

GitHub Outage Tracker: Is GitHub Cooked?

isgithubcooked.com

Post reply on HN
Print view View original

81–90 of 195 posts

Re: GitHub Outage Tracker: Is GitHub Cooked?

#81

Post by majormajor »

joshuahedlund wrote: post #57

Given that the outages are caused by record traffic, so far they are only cooked in the Yogi Berra sense of “Nobody goes there anymore, it’s too crowded”

Random AI slop code pushers don't pay the bills. Traffic isn't revenue.

I've started suggesting to my CTO that we move to something else because we do pay the Github bills and yet we are also affected by other people's activity. Avoiding that is not a hard engineering problem for an alternative.

A world where a record amount of code is pushed to GH but a large amount of paying customers leave would be "cooked"

Re: GitHub Outage Tracker: Is GitHub Cooked?

#82

Post by madrox »

kashnote wrote: post #12

I think we need to have a little more sympathy for GitHub. You could justify the jabs when we could all blame any outage on the migration to Azure, but then they shared numbers around the scale they're dealing with now that everyone is constantly building and pushing with AI. I think it's commendable that they're not limiting access to the site or (intentionally) throttling newcomers. Yes, they need to get this figur…

I have a lot of sympathy for the people working at GitHub trying to keep it going, but I do not have sympathy for GitHub. That's what they're paid for. I have things to get done, and it has no sympathy for me.

Re: GitHub Outage Tracker: Is GitHub Cooked?

#83

Post by bluefirebrand »

Bratmon wrote: post #74
joshuahedlund wrote: post #57

Given that the outages are caused by record traffic, so far they are only cooked in the Yogi Berra sense of “Nobody goes there anymore, it’s too crowded”

Many people think we're on the path to having separate webs for AIs and humans. This may be the beginning of that.

As much as I would like that to happen, I don't think it is at all possible until we solve the "is a human" problem.

Re: GitHub Outage Tracker: Is GitHub Cooked?

#84

Post by Waterluvian »

joshuahedlund wrote: post #57

Given that the outages are caused by record traffic, so far they are only cooked in the Yogi Berra sense of “Nobody goes there anymore, it’s too crowded”

Gets me wondering… if I get used to the fact that some location is busy, I stop going there. Other people might do the same. Could this create a detectable wavy signal over time? People stop going because it’s busy. It stops being busy. People go because it’s not busy. Repeat. Is this a known phenomenon?

Re: GitHub Outage Tracker: Is GitHub Cooked?

#85

Post by logicchains »

tyre wrote: post #58
gogobio wrote: post #23

Earlier quoted context omitted.

Sympathy? It's a Microsoft company that is being ran with a consistency of a startup in early seed rounds. Their downtime is abhorrent and unacceptable as far as enterprise goes. Their engineers look like absolute amateurs allowing for such low class work it results in their customers experiencing industry leading downtime.

This is uncharitable, rude, and pretty baseless. Unless you have a lot of direct, personal information about GitHub engineers, they’re dealing with a huge spike in traffic. Is their uptime acceptable? No. But personal attacks aren’t necessary or constructive.

How low would uptime have to get before you'd consider holding the engineers personally responsible for it?

Re: GitHub Outage Tracker: Is GitHub Cooked?

#86

Post by CoastalCoder »

tryauuum wrote: post #68
californical wrote: post #61

Earlier quoted context omitted.

If I own a restaurant and buy bread from a supplier — BreadHub. And 99% of the bread that I get is good but 1% of the loaves, they forgot to add flour. Consistently, for years, they always have loaves missing a key ingredient that I still end up paying for. I can be pretty sure that BreadHub have a pretty major internal issue, and should probably be questioning their competence, regardless of their “scale”, and witho…

Please don't do it. Now somebody will come and will try to improve your bread analogy. We will be discussing bread for eons

Okay, let's use a car analogy...

Re: GitHub Outage Tracker: Is GitHub Cooked?

#87

Post by rpearl »

kashnote wrote: post #12

I think we need to have a little more sympathy for GitHub. You could justify the jabs when we could all blame any outage on the migration to Azure, but then they shared numbers around the scale they're dealing with now that everyone is constantly building and pushing with AI. I think it's commendable that they're not limiting access to the site or (intentionally) throttling newcomers. Yes, they need to get this figur…

If you do the math, their numbers are something like 1000 commits per second. That's not pushes per second either, it's just the total number of commits summed up over a week, not all of which trigger much of anything to happen.

Yes, that is scale. And yes, that's not actual requests per second. But it's the sort of scale that big (and even mid-sized) tech has known how to deal with for decades. Microsoft doesn't have an excuse.

Re: GitHub Outage Tracker: Is GitHub Cooked?

#88

Post by grumple »

gogobio wrote: post #23
kashnote wrote: post #12

I think we need to have a little more sympathy for GitHub. You could justify the jabs when we could all blame any outage on the migration to Azure, but then they shared numbers around the scale they're dealing with now that everyone is constantly building and pushing with AI. I think it's commendable that they're not limiting access to the site or (intentionally) throttling newcomers. Yes, they need to get this figur…

Sympathy? It's a Microsoft company that is being ran with a consistency of a startup in early seed rounds. Their downtime is abhorrent and unacceptable as far as enterprise goes. Their engineers look like absolute amateurs allowing for such low class work it results in their customers experiencing industry leading downtime.

Typically when a company gets acquired, they get thrown a bunch of terrible initiatives like “change your cloud provider” that distract from their mission, executives saying “more AI!” and they definitely don’t hear anyone in management say “prioritize technical debt”.

Re: GitHub Outage Tracker: Is GitHub Cooked?

#89

Post by ChrisMarshallNY »

kashnote wrote: post #12

I think we need to have a little more sympathy for GitHub. You could justify the jabs when we could all blame any outage on the migration to Azure, but then they shared numbers around the scale they're dealing with now that everyone is constantly building and pushing with AI. I think it's commendable that they're not limiting access to the site or (intentionally) throttling newcomers. Yes, they need to get this figur…

I strongly suspect that quite a few HN members are busy creating agents that are furiously stuffing GH.

As I said before (and it was not a popular comment), it's easy to be an "armchair quarterback," with these services, as I think the brittleness was already baked in, and just waiting for the right time to crack. The only true way to have a robust platform, is to design something that will scale, from the start, and many startups don't do that, because they are feverishly trying to get out an MVP; even if it is a mess of bubblegum and baling wire.

They always say "We'll get it done right, once we get funding," but that never happens.

But my sympathy is limited by the fact that MS paid a lot of money for this, quite a while ago, and that is one company that knows all about issues of scale. They should have seen this coming.

Re: GitHub Outage Tracker: Is GitHub Cooked?

#90

Post by vrosas »

logicchains wrote: post #85
tyre wrote: post #58

Earlier quoted context omitted.

This is uncharitable, rude, and pretty baseless. Unless you have a lot of direct, personal information about GitHub engineers, they’re dealing with a huge spike in traffic. Is their uptime acceptable? No. But personal attacks aren’t necessary or constructive.

How low would uptime have to get before you'd consider holding the engineers personally responsible for it?

Agreed. At some point to have to stop hugging your ops team and start firing them.
Post reply on HN