Someone said GitHub is racing to the mythical "zero nines of availability" and I love it
Hmm... 88.8888888%?
Jesus, that's both horrible and seems within reach.
Incident with Actions and Pages
61–70 of 407 posts
Re: Incident with Actions and Pages
#62My action failed with "Unexpected error fetching GitHub release for tag refs/heads/master: HttpError: Sorry. Your account was suspended" Which certainly made me shit myself, briefly.
Same. It's weird how I always find out that GitHub is down before GitHub does. Took 15 minutes before it appeared on githubstatus.com
All these monitoring rules are of the format "when 500 errors > baseline for x minutes". Otherwise you'd have monitoring alerts every second. So it is normal for users to already see errors before github officially counts it as an outage.
Re: Incident with Actions and Pages
#63My action failed with "Unexpected error fetching GitHub release for tag refs/heads/master: HttpError: Sorry. Your account was suspended" Which certainly made me shit myself, briefly.
Same. It's weird how I always find out that GitHub is down before GitHub does. Took 15 minutes before it appeared on githubstatus.com
More likely that 'update the Status site' lives a long way down their incident response plan, and they have alarms going off well before that
Re: Incident with Actions and Pages
#64'Degraded' should be banned in status pages. It sounds just irresponsible, like "Yeah, it can be slow or something sometime. Whatever. Who cares"
Straight-up, "degraded" should strictly mean "may be slower, or so slow it randomly fails" on these kinds of status pages.
Re: Incident with Actions and Pages
#65Are there any GitHub Actions-compatible CI services out there that don't rely on their infrastructure? I know of depot's but no others; are these resilient to these outages or do they still lose functionality? I imagine the latter but I don't know.
We currently use external runners (Blacksmith.sh), but that didn't shield us from this as GitHub actions is still the control plane for triggering and monitoring them.
We're now considering Buildkite (apparently they have a GH actions migration tool) or self hosting something (GitLab CI, maybe even Jenkins), as it looks like that would've kept ticking over since we're still seeing webhooks being triggered today during the downtime.
Re: Incident with Actions and Pages
#66Whilst you're waiting for it to come back, try out AGENT-CI (which is a project I built.), which runs GitHub Actions on your machine: https://agent-ci.dev . (Open source, etc.) No, it's not like "act," because it uses the standard Github runner, the difference is that the control plane is an emulation of api.github.com, because of this we can do all kinds of nice things: Caching in ~0 ms. Pause on failure, so you can…
What I don’t get about this is how you run OS specific tasks (Windows, macOS, Linux)..
I started playing with proxmox VMs and containers in them (docker and tart) to see if I can build some local infrastructure to properly solve this…
Re: Incident with Actions and Pages
#67Wasn’t my fault this time! I haven’t started work yet. https://news.ycombinator.com/item?id=47237377
Hah, I know the feeling. I installed Ubuntu on a PC recently, it obviously happened to be one of the days they got DDOSed and apt repos were unreachable. I had other things to take care of, so I put it aside for the next week or so. It didn't help very much, cause after picking it back up, halfway through, Snapcraft went down.
Re: Incident with Actions and Pages
#68Too many times we've been bitten by this - it has been an issue too many times to count.
This is why we don't use Github Actions, kids.
Seriously, its a proprietary build service that puts the keys to the kingdom in someone elses' control. Just: No!
Print this status page to PDF so you've got it handy next time someone castigates you for not using Github Actions, folks.
Re: Incident with Actions and Pages
#69Yeah I'm getting an error where it says account has been suspended. They really are becoming an embarassment
Re: Incident with Actions and Pages
#70Are there any GitHub Actions-compatible CI services out there that don't rely on their infrastructure? I know of depot's but no others; are these resilient to these outages or do they still lose functionality? I imagine the latter but I don't know.
We currently use external runners (Blacksmith.sh), but that didn't shield us from this as GitHub actions is still the control plane for triggering and monitoring them. We're now considering Buildkite (apparently they have a GH actions migration tool) or self hosting something (GitLab CI, maybe even Jenkins), as it looks like that would've kept ticking over since we're still seeing webhooks being triggered today durin…
Try Depot CI as well. Supports a GHA syntax but the entire control plane is ours with our own engine.