It's so weird because github used to be known for rock solid stability and now the entire reputation has changed.
Incident with Actions and Pages
111–120 of 407 posts
Re: Incident with Actions and Pages
#112lol
Re: Incident with Actions and Pages
#113Earlier quoted context omitted.
All these monitoring rules are of the format "when 500 errors > baseline for x minutes". Otherwise you'd have monitoring alerts every second. So it is normal for users to already see errors before github officially counts it as an outage.
In a high performance service with good maintenance and upkeep, you page for all 500s. A noisy pager forces the team to fix the 500s. Maybe the Github Actions infrastructure isn't run like that. edit: my oncall rotation notified on all 500s, 24/7, not just rates - https://news.ycombinator.com/item?id=48279262
Im curious about this: because in my experience (working on smaller services though), a small number of errors is always there, as a "baseline".
Recently there was this: https://news.ycombinator.com/item?id=47252971 "10% of Firefox crashes are caused by bitflips"
Which makes me think a small amount of random issues which happen even though nothing is broken, is normal everywhere. Especially once move things around on a network, there's potential for a lot more random errors.
Re: Incident with Actions and Pages
#114Re: Incident with Actions and Pages
#115Earlier quoted context omitted.
All these monitoring rules are of the format "when 500 errors > baseline for x minutes". Otherwise you'd have monitoring alerts every second. So it is normal for users to already see errors before github officially counts it as an outage.
In a high performance service with good maintenance and upkeep, you page for all 500s. A noisy pager forces the team to fix the 500s. Maybe the Github Actions infrastructure isn't run like that. edit: my oncall rotation notified on all 500s, 24/7, not just rates - https://news.ycombinator.com/item?id=48279262
that is absolutely not the case for any system of size and scale. that would just burn out the on-call team and not result in improvements. Error rates/budgets are used instead.
Re: Incident with Actions and Pages
#116Earlier quoted context omitted.
No, it doesn’t. Their competition is not similarly unstable, despite existing in the same world of LLMs. Think critically.
Devil’s advocate, Pareto heuristic would let us speculate that 80% of LLM traffic would be aimed directly at the largest provider, i.e. GitHub.
I think it’s much more than 80%, it’s probably the default recommendation and folks who aren’t technical would just accept it. Probably closer to 95% or more
Re: Incident with Actions and Pages
#117Are there any GitHub Actions-compatible CI services out there that don't rely on their infrastructure? I know of depot's but no others; are these resilient to these outages or do they still lose functionality? I imagine the latter but I don't know.
github actions themselves can be self hosted, its quite nice actually to be able to keep your same patterns as cloud hosted actions and with one line change to the yaml have it running on your own hardware. I do this for actions that take 6-7 hours so I am not burning through the 3000 minutes that come free with my account.
what kind of actions take that long? some kind of compilation task / gigantic test suite ala SQLite?
Re: Incident with Actions and Pages
#118Why do they go down so often? Is it true that the reason is that they've incorporated too much AI without human review?
I personally trigger github actions approximately 50x more than I did prior to AI-driven developer coding and I'm not alone.
Okay so the recent outages are also likely due to increased load due to AI assisted development speeding up workflows.
Re: Incident with Actions and Pages
#119Shout out to all my SF 5am crew checking if their overnight prs passed CI. Real 597 “member of technical staff” energy. I guess we should expect this, it is a Tuesday!
Re: Incident with Actions and Pages
#120Earlier quoted context omitted.
GitHub had a blog post about this recently. They reported a significant uptick in volume (repos created, PRs, etc.), which they attribute to AI usage and tooling.
Do you really believe their competition hasn’t seen the same increase? Because their competition certainly hasn’t seen the same instability issues.
This plus in a well-designed system an increase in load might cause new jobs to stop running but shouldn't take down the whole system.