Earlier quoted context omitted.
They can't really cite the situation as a problem given their hand in creating and continuing it.
The sysadmins didn't make any of those decisions.
I suppose the idiocy of their parent company is their job security.
Incident with Issues and Webhooks – Resolved
71–80 of 275 posts
Re: Incident with Issues and Webhooks – Resolved
#72Github has published some incredible usage rate increase numbers, which they ascribe to the rise of agentic coding. At some point, they are going to have to change rate limits, cut free-tier usage, or find some other path to reducing load. It's clear that their infrastructure can't keep up with this significant increase, and it's unlikely that they're going to just absorb the increased costs themselves. Very curious…
Isn't the data that flows through Github so valuable that they (Microsoft) are happy to eat the cost? I don't have a clear idea how that value can be captured, since it's going to be 90% AI generated code that anyone can scrape (public projects) or can't be used (private projects), so perhaps you're right.
Its easier for them to scrape than it is for anyone else. they also have a lot more meta data about the code which may be useful.
Do Github terms entirely prevent them from making use of data in private projects.
Re: Incident with Issues and Webhooks – Resolved
#73Github has published some incredible usage rate increase numbers, which they ascribe to the rise of agentic coding. At some point, they are going to have to change rate limits, cut free-tier usage, or find some other path to reducing load. It's clear that their infrastructure can't keep up with this significant increase, and it's unlikely that they're going to just absorb the increased costs themselves. Very curious…
For literally decades, I’ve observed that there are systems that make each operation cheap and systems that work hard to scale out. The former frequently seems to wildly outperform the latter. GitHub, for example, seems to implement the main repository /pulls page as a search query, which is hinted at by the prefilled search bar and was mostly confirmed last week when the search backend failed and pull requests didn’…
Are you telling me you don’t want a chat interface to greet you when you log in to GitHub?
Re: Incident with Issues and Webhooks – Resolved
#74This is reaching an unacceptable level of performance. There isn't a week that work isn't interrupted by GH.
AI agents have changed the scalability properties of basically the entire internet. It used to be that GitHub could rely on a finite number of people interacting with their platform in real human ways in real observable patterns. So I'm assuming that they scale for those patterns, and optimize for the UI and UX hotspots. But now everyone's got a moltbot running 24/7, sometimes many, and it's completely overloading a…
[dead]
Re: Incident with Issues and Webhooks – Resolved
#75Earlier quoted context omitted.
For literally decades, I’ve observed that there are systems that make each operation cheap and systems that work hard to scale out. The former frequently seems to wildly outperform the latter. GitHub, for example, seems to implement the main repository /pulls page as a search query, which is hinted at by the prefilled search bar and was mostly confirmed last week when the search backend failed and pull requests didn’…
I think you need to broaden your focus here - I can't really remember any significant downtime before the Microsoft acquisition and the data supports my memories. Microsoft bought Github and migrated to Azure, which is explains the findings. The query performance was fine before they started serving from Azure. I mean honestly, as though there isn't one single person competent enough to read some logs and horizontall…
> I think you need to broaden your focus here - I can't really remember any significant downtime before the Microsoft acquisition and the data supports my memories.
This is the opposite of my recollection, actually. I distinctly remember having conversations about Github struggling to scale well before MS was involved, and people claiming that MS had somehow saved Github because it had stabilized and begun adding features again.
> The query performance was fine before they started serving from Azure.
This may be correct though. The Azure migration seems more aligned with the timeline of struggling to scale.
Re: Incident with Issues and Webhooks – Resolved
#76It feels weird (sad?) that I'm starting to get a sixth sense for when Github is going to a service disruption. About an hour ago, clicking "Resolve Conversation" in a Pull Request failed a few times with an error message that appeared lower on the page (outside the viewport), and which I did not see the first few times. I had to reload the page after every few actions to get the server to register new ones. I told a…
Literally just had this same signal with PR review comments. Checked the status page, saw it was green, and (correctly) assumed “not for long!”
Re: Incident with Issues and Webhooks – Resolved
#77Github has published some incredible usage rate increase numbers, which they ascribe to the rise of agentic coding. At some point, they are going to have to change rate limits, cut free-tier usage, or find some other path to reducing load. It's clear that their infrastructure can't keep up with this significant increase, and it's unlikely that they're going to just absorb the increased costs themselves. Very curious…
From the GitHub COO on April 3rd:
Platform activity is surging. There were 1 billion commits in 2025.
Now, it's 275 million per week, on pace for 14 billion this year if
growth remains linear (spoiler: it won't.)
GitHub Actions has grown from 500M minutes/week in 2023 to 1B minutes/week
in 2025, and now 2.1B minutes so far this week.
So we're pushing incredibly hard on more CPUs, scaling services, and
strengthening GitHub’s core features.
They also had a recent blog post about availability: https://github.blog/news-insights/company-news/an-update-on-...
I don't envy the scaling issues the GitHub engineers are facing! #HugOps
Re: Incident with Issues and Webhooks – Resolved
#78Github has published some incredible usage rate increase numbers, which they ascribe to the rise of agentic coding. At some point, they are going to have to change rate limits, cut free-tier usage, or find some other path to reducing load. It's clear that their infrastructure can't keep up with this significant increase, and it's unlikely that they're going to just absorb the increased costs themselves. Very curious…
Can you share where did they published that?
Re: Incident with Issues and Webhooks – Resolved
#79At this point we can just have a Bot repost this exact submission every day and it would be right more often than not.
It exists, but it runs as a github action, so it's not working right now.
Re: Incident with Issues and Webhooks – Resolved
#80It feels weird (sad?) that I'm starting to get a sixth sense for when Github is going to a service disruption. About an hour ago, clicking "Resolve Conversation" in a Pull Request failed a few times with an error message that appeared lower on the page (outside the viewport), and which I did not see the first few times. I had to reload the page after every few actions to get the server to register new ones. I told a…
[flagged]