Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Incident with Pull Requests, Issues, Git Operations and API Requests

githubstatus.com

Post reply on HN
Print view View original

171–180 of 221 posts

Re: Incident with Pull Requests, Issues, Git Operations and API Requests

#171

Post by gen220 »

05hundred wrote: post #72
elzbardico wrote: post #16

Earlier quoted context omitted.

GH was acquired by microsoft some eight years ago. It has been working quite well until recently. People may have had complaints about functionality, features, commercial issues, but the thing used to at least have a decent uptime until recently.

> It has been working quite well until recently. I'm not sure how reliable the data is, but average uptime seems to have dipped measurably starting within a year of the aquisition, according to https://damrnelson.github.io/github-historical-uptime/

FWIW, I'm not convinced that chart is necessarily an accurate representation of pre-acquisition reality. It would really surprise me if GitHub did not have a single sev-0 pre-acquisition, but it wouldn't surprise me if they were not formally captured and reported in a format that would make its way into their current status page's database.

Re: Incident with Pull Requests, Issues, Git Operations and API Requests

#173

Post by mhitza »

spaceman_2020 wrote: post #18

is it me or ever since AI coding became the norm, there have been way more outages with otherwise reliable services? I get downtime on Supabase every few weeks. Even Cloudflare. And now Github

Enterprise software and support services have consideraby (in my personal experience) degraded over the last year.

Less care on process, or quality, more focus on "just ship".

I've also seen and heard from peers this happening in multiple smaller outsorcing companies.

Re: Incident with Pull Requests, Issues, Git Operations and API Requests

#174

Post by IshKebab »

ricardbejarano wrote: post #14
EduardoBautista wrote: post #7

Earlier quoted context omitted.

I'd consider self-hosting GitHub Enterprise before putting my team through the pain of Gitlab.

How is it painful to use GitLab? Curious, as a user of both, I find them both nice. I like GitLab CI/CD more than I do GHA, but that's personal preference/bias more than anything objective.

Gitlab is pretty decent. Honestly I would say there's not much between GitHub and Gitlab. Gitlab's CI is more powerful than GitHub's IMO, but the UX is a bit worse. But it's really marginal.

They're both slow and have tons of long-standing missing features. But they're ok. I'd definitely rather use Gitea/Gogs/Forgejo, and maybe Tangled if it supported normal setups (e.g. private repos).

Re: Incident with Pull Requests, Issues, Git Operations and API Requests

#175

Post by pocksuppet »

OkayPhysicist wrote: post #158
pocksuppet wrote: post #125

Earlier quoted context omitted.

You don't get to control that. It is Microsoft's right to do whatever it wants with GitHub - it could shut down tomorrow, or demand face ID. If you want to control what happens with a thing, you have to make the thing instead of letting someone else make the thing and sell it to Microsoft. Your choice is to accept the product that exists on the market, switch to another product that exists on the market (such as Code…

You missed the option to "lobby the government to tell Microsoft to straighten up and fly right". Microsoft is a corporation, a legal entity only created with the permission and grace of the State of Washington.

Sure, I guess skywriting prayers to the spaghetti monster is also an option, but neither of those will do anything.

Re: Incident with Pull Requests, Issues, Git Operations and API Requests

#176

Post by keeda »

bushbaba wrote: post #142
hbn wrote: post #129

Earlier quoted context omitted.

Commits are up 14x year-over-year https://x.com/kdaigle/status/2040164759836778878

Not a valid excuse without knowing what their historical growth rate has been. And how much of the instability is load related.

GitHub has been publishing their growth numbers since at least 2016: https://octoverse.github.com/2016/

However, they have reported numbers along rather inconsistent dimensions. Like, historically they've focused on number of repos and users and later PR's and issues, and often catch-all terms like "contributions" which includes all of those + comments etc... but the number of commits alone (which apparently is the main culprit now?) has been mentioned very sporadically. This has made it hard to get a consistent sense of historical growth.

Without any other information, however, it is reasonable to assume that a 14x in commits is the prime candidate for instability. Especially since commits are write traffic, which is much harder to scale than read traffic. Plus every 3 - 5x increase in scale can reveal bottlenecks in your distributed systems that you never knew existed, so they probably have like 2 - 3 "generations" of bottlenecks to figure out!

Re: Incident with Pull Requests, Issues, Git Operations and API Requests

#177

Post by gen220 »

rsyring wrote: post #159
gen220 wrote: post #6

https://isgithubcooked.com Normally I defend GH in the comments of these incidents but it’s been an impressively bad month by their standards, even when you filter for critical components filter out sev-2’s and 3’s.

Of all the sites/graphs I've seen of GH outages, this one is the most striking IMO: https://damrnelson.github.io/github-historical-uptime/ Unfortunately, it doesn't look like it's being updated with new data. But it wouldn't look any better for GH if it was.

FWIW, I'm not convinced that chart is necessarily an accurate representation of pre-acquisition reality. It would really surprise me if GitHub did not have a single sev-0 pre-acquisition, but it wouldn't surprise me if they were not formally captured and reported in a format that would make its way into their current status page's database.

Re: Incident with Pull Requests, Issues, Git Operations and API Requests

#178

Post by voncheese »

chris_money202 wrote: post #133
voncheese wrote: post #87

Earlier quoted context omitted.

Yeah, that and Microsoft has been slow to move the infrastructure to something that scales better to handle that load. The more surpassing part is that Microsoft hasn't figured out a way to manage/contain the AI-sourced traffic better so it doesn't create all this noisy neighbor problems for non-AI usage/users.

Github's core platform doesn't really make that separation, anything a human can leverage on github an AI agent can as well, just faster and with heavier usage. End of day agents and humans are using the same services.

Sure, still need to enable access the same info but feels like bucketing the clients into

bucket1 = clients that were working just fine before (users and whatever automation they had in place) bucket2 = ai clients that contributed to, if not flat out caused, the scale problems

then slowing down/limiting the bucket2 clients while keeping the bucket1 clients rolling as-is, is both doable and keeps existing customers happy while the underlying infra gets scale/perf improvements needed to support ai clients at scale.

Re: Incident with Pull Requests, Issues, Git Operations and API Requests

#179

Post by gred »

thr0w4w4y1337 wrote: post #152
gred wrote: post #150

New PR: revert GitHub software and infrastructure to version of June 1st, 2018. New PR: disable new user signups for 6 months HR initiative: all future KPIs automatically require three-nines availability; all bonuses are forfeited, regardless of accomplishments, if annual availability falls below target HR initiative: fire CEO and CTO

Github does not have a CEO

We're halfway there!
Post reply on HN