Looks like they really want to get a PR deployed, but there's still not enough duct tape on it.
GitHub incident 2022-03-23
11–20 of 206 posts
Re: GitHub incident 2022-03-23
#122nd day in a row isn’t it?
And 6 days ago: https://news.ycombinator.com/item?id=30711269
Re: GitHub incident 2022-03-23
#13This has been a pretty rough week for GitHub
Re: GitHub incident 2022-03-23
#14It seems like we haven't had a non-robot status update on the status page in days since this what seems like daily occurrence. I figure at this point we'd get something of why this is happening. I also don't appreciate our builds freezing, unable to be cancelled and then eating up hundreds of minutes.
Billing should always be built on a "ping" IMO and not start/stop hooks. The latter is shockingly bad for customers during times of unreliability. The former sounds stupid and requires more infrastructure from the one offering the service, but I think it's more fair.
I haven't used GA in a way where it actually costed me anything, but having minutes just tick away while you can't do anything is really stupid if that's the case.
Edit: Another sane solution would probably be to record outage periods and have Billing automatically reconcile for every customer when invoicing. This would require them to admit the outage durations however, so it may be flawed from a human perspective.
Re: GitHub incident 2022-03-23
#15It seems like we haven't had a non-robot status update on the status page in days since this what seems like daily occurrence. I figure at this point we'd get something of why this is happening. I also don't appreciate our builds freezing, unable to be cancelled and then eating up hundreds of minutes.
Do they give you the minutes back if there's an incident during the period where a job is running?
Re: GitHub incident 2022-03-23
#16https://www.githubstatus.com/history
21 incident outages in just 3 months. At this rate the benefits of running your own gitea or gitlab are starting to become competitive.
Re: GitHub incident 2022-03-23
#17It seems like we haven't had a non-robot status update on the status page in days since this what seems like daily occurrence. I figure at this point we'd get something of why this is happening. I also don't appreciate our builds freezing, unable to be cancelled and then eating up hundreds of minutes.
Do they give you the minutes back if there's an incident during the period where a job is running?
You will have to contact them for them to credit you, that's what we did
Re: GitHub incident 2022-03-23
#18Earlier quoted context omitted.
Do they give you the minutes back if there's an incident during the period where a job is running?
You will have to contact them for them to credit you, that's what we did
This is totally unsurprising and also totally unacceptable IMO. They should automatically wipe out all build minute usage during outages for every account if they insist on architecting their system in this way.
Re: GitHub incident 2022-03-23
#19Again? Last time that happened was 24 hours ago? [0] It is really getting unreliably bad. Like I said before, having a self-hosted backup seems to make more sense.
Re: GitHub incident 2022-03-23
#20I hope they figure out what’s going on every morning. Heard from inside they don’t know why the db dies everyday but restarting it fixes it.