Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

GitHub Outage Tracker: Is GitHub Cooked?

isgithubcooked.com

Post reply on HN
Print view View original

151–160 of 195 posts

Re: GitHub Outage Tracker: Is GitHub Cooked?

#151

Post by mejutoco »

tyre wrote: post #58
gogobio wrote: post #23

Earlier quoted context omitted.

Sympathy? It's a Microsoft company that is being ran with a consistency of a startup in early seed rounds. Their downtime is abhorrent and unacceptable as far as enterprise goes. Their engineers look like absolute amateurs allowing for such low class work it results in their customers experiencing industry leading downtime.

This is uncharitable, rude, and pretty baseless. Unless you have a lot of direct, personal information about GitHub engineers, they’re dealing with a huge spike in traffic. Is their uptime acceptable? No. But personal attacks aren’t necessary or constructive.

Is it a personal attack if it is not directed to a person?

Re: GitHub Outage Tracker: Is GitHub Cooked?

#152

Post by flohofwoe »

jeremyjh wrote: post #98
gogobio wrote: post #23

Earlier quoted context omitted.

Sympathy? It's a Microsoft company that is being ran with a consistency of a startup in early seed rounds. Their downtime is abhorrent and unacceptable as far as enterprise goes. Their engineers look like absolute amateurs allowing for such low class work it results in their customers experiencing industry leading downtime.

If you haven't read this, you really have no idea what they are dealing with. https://cursor.com/blog/git-at-any-scale

They could throttle git operations to a level which isn't noticeable by normal 'human' usage but seriously slows down automated usage. These are basically DDOS attacks.

Re: GitHub Outage Tracker: Is GitHub Cooked?

#153

Post by DANmode »

tyre wrote: post #58
gogobio wrote: post #23

Earlier quoted context omitted.

Sympathy? It's a Microsoft company that is being ran with a consistency of a startup in early seed rounds. Their downtime is abhorrent and unacceptable as far as enterprise goes. Their engineers look like absolute amateurs allowing for such low class work it results in their customers experiencing industry leading downtime.

This is uncharitable, rude, and pretty baseless. Unless you have a lot of direct, personal information about GitHub engineers, they’re dealing with a huge spike in traffic. Is their uptime acceptable? No. But personal attacks aren’t necessary or constructive.

Why do I feel like GitHub, without the Microsoft tax, would handle this huge spike in traffic fine?

Can you comment on that?

Re: GitHub Outage Tracker: Is GitHub Cooked?

#154

Post by flohofwoe »

chris_money202 wrote: post #104
cr125rider wrote: post #91

Earlier quoted context omitted.

That seems a bit crass but the underlying sentiment stands. Microsoft has more money than God. 4T valuation. When you say you’re worth that much, no excuses. Figure it out.

So your proposal is for Microsoft to invest their free cash in helping support a bunch of AI coders maintain their pet projects? Sounds like you should be a CEO!

You don't need to be a CEO to come to the conclusion that throttling to discourage excessive usage is probably a good idea.

Re: GitHub Outage Tracker: Is GitHub Cooked?

#155

Post by DANmode »

chris_money202 wrote: post #104
cr125rider wrote: post #91

Earlier quoted context omitted.

That seems a bit crass but the underlying sentiment stands. Microsoft has more money than God. 4T valuation. When you say you’re worth that much, no excuses. Figure it out.

So your proposal is for Microsoft to invest their free cash in helping support a bunch of AI coders maintain their pet projects? Sounds like you should be a CEO!

The move would have been to avoid purchasing and operating a subsidiary wit a question mark for a budget,

unless the firm was prepared to see it through.

Re: GitHub Outage Tracker: Is GitHub Cooked?

#156

Post by adithyassekhar »

kashnote wrote: post #12

I think we need to have a little more sympathy for GitHub. You could justify the jabs when we could all blame any outage on the migration to Azure, but then they shared numbers around the scale they're dealing with now that everyone is constantly building and pushing with AI. I think it's commendable that they're not limiting access to the site or (intentionally) throttling newcomers. Yes, they need to get this figur…

I’ve noticed there is 5000 api call limit on gh cli. Haven’t looked at whether it was already there or when does it reset etc. good for them finally if they weren’t already doing that.

Re: GitHub Outage Tracker: Is GitHub Cooked?

#157

Post by DANmode »

vrosas wrote: post #92
rpearl wrote: post #87

Earlier quoted context omitted.

If you do the math, their numbers are something like 1000 commits per second. That's not pushes per second either, it's just the total number of commits summed up over a week, not all of which trigger much of anything to happen. Yes, that is scale. And yes, that's not actual requests per second. But it's the sort of scale that big (and even mid-sized) tech has known how to deal with for decades. Microsoft doesn't hav…

Their excuse is called Azure and if you’ve ever had to deal with it you’d know it’s an amazing(ly awful) excuse to have.

Free GitHub, let the engineers choose their own infra.

Bit of a Clownstrike this has become!

Re: GitHub Outage Tracker: Is GitHub Cooked?

#158

Post by selfhoster1312 »

firesteelrain wrote: post #133
hansvm wrote: post #101

Earlier quoted context omitted.

The phrasing could have been better, but we've talked as a team about cancelling a $100k/yr contract nearly half a dozen times the last year, and the only thing keeping us from doing so is various "compliance" issues, and to a lesser degree the friction from physically moving. That's a tenuous moat, and if I were a GH PM I'd probably want to know that the current instability is somewhere near critical mass.

You are certainly free to attempt to migrate your workloads to your own or leased data center space in your area then buy your own on prem cloud like Dell Private Cloud and deal with all of the maintenance yourself. I don’t think it would amount to $100k/yr and you would be taking on all the risk.

For most teams selfhosting on 1U is more than enough which would cost about 1200€/year for housing + let's say 1800€/year for hardware acquisition/depreciation (probably more like 300€/year if you can work with second hand hardware) so that's about 3000€/year or 97% reduction.

Of course you may not want to manage the hardware yourself in which case you'd go for a dedicated server from a reputable provider (i.e. not microsoft/google) like OVH's RISE-XL, Hetzner's EX131, Scaleway's Core-9-L (or an equivalent solution from a smaller provider) for about 4000€/year.

Now let's say you add 8k€/year donations to your distro of choice, forge of choice, and other FLOSS projects and 8k€/year for backups. You got the whole thing running sustainably with >99.9% uptime for 20k€/year or 20% of the original bill.

Re: GitHub Outage Tracker: Is GitHub Cooked?

#159

Post by selfhoster1312 »

tyre wrote: post #64
californical wrote: post #61

Earlier quoted context omitted.

If I own a restaurant and buy bread from a supplier — BreadHub. And 99% of the bread that I get is good but 1% of the loaves, they forgot to add flour. Consistently, for years, they always have loaves missing a key ingredient that I still end up paying for. I can be pretty sure that BreadHub have a pretty major internal issue, and should probably be questioning their competence, regardless of their “scale”, and witho…

I’m not saying users don’t have a right to be pissed or aren’t justified in looking at other options. I’m saying that GH is operating at a huge scale with (probably) lots of technical debt and a forced migration to new infrastructure. I would be (and am) highly critical of leadership. I’m not going to make strong assertions about ICs without knowing their context. I’ve worked at a company with a sterling reputation f…

I mean, the leadership brought it on themselves. For a few years they encouraged sloperating Copilot. Before that, they were encouraging Github Actions usage.

Now the scale problems they're facing is because of those 2 actions of theirs. They didn't have to encourage such growth to turn a profit, and they still don't have to provide gratis services to slopware bots providing 0 value for society.

Re: GitHub Outage Tracker: Is GitHub Cooked?

#160

Post by selfhoster1312 »

Freedom2 wrote: post #120
californical wrote: post #61

Earlier quoted context omitted.

If I own a restaurant and buy bread from a supplier — BreadHub. And 99% of the bread that I get is good but 1% of the loaves, they forgot to add flour. Consistently, for years, they always have loaves missing a key ingredient that I still end up paying for. I can be pretty sure that BreadHub have a pretty major internal issue, and should probably be questioning their competence, regardless of their “scale”, and witho…

I don't think this is an apt analogy. The bread makeup is still there, as far as I'm aware, no users have lost any data or are missing "key" ingredients. An outage is more like a shipping issue with the supplier, if it's owned wholly by them.

Are you saying the dough is fine but 1% of the batch wasn't cooked? Or rather, that 1% of batches isn't cooked, taking your entire downstream activity down until a new batch arrives?
Post reply on HN