Incident with Actions and Codespaces

#1

Post by saltymimir » Wed, Oct 05, 2022, 1:48 PM UTC

Incident with Actions and Codespaces
githubstatus.com

Re: Incident with Actions and Codespaces

#2

Post by nathankleyn » Wed, Oct 05, 2022, 3:07 PM UTC

Actions and Codespaces are listed as having "degraded performance" now too - seeing a lot of action builds disappear into the ether and never get added to the queue to run.

Re: Incident with Actions and Codespaces

#3

Post by derpsteb » Wed, Oct 05, 2022, 3:11 PM UTC

nathankleyn wrote: post #2

Actions and Codespaces are listed as having "degraded performance" now too - seeing a lot of action builds disappear into the ether and never get added to the queue to run.

5mins later: WAIT, YOU FORGOT ALL THESE 20 RUNS!!!!

Re: Incident with Actions and Codespaces

#4

Post by codechecker4 » Wed, Oct 05, 2022, 4:05 PM UTC

derpsteb wrote: post #3

nathankleyn wrote: post #2

Actions and Codespaces are listed as having "degraded performance" now too - seeing a lot of action builds disappear into the ether and never get added to the queue to run.

5mins later: WAIT, YOU FORGOT ALL THESE 20 RUNS!!!!

Yeah 20 or so runs. And I'm like: well they did say added to the queue successfully.

Re: Incident with Actions and Codespaces

#5

Post by javierluraschi » Wed, Oct 05, 2022, 4:50 PM UTC

10min troubleshoot time * 10K devs troubleshooting GitHub Actions right now... there goes one year for comunal dev productivity.

Re: Incident with Actions and Codespaces

#6

Post by fasteddie31003 » Wed, Oct 05, 2022, 4:58 PM UTC

I'm working on a proactive 3rd party downtime detection tool called AwareOps. Here is some of the recent data from Github's incidents https://app.awareops.com/whatisdown/orgs/437dc8b8-2386-4ef5-... .

Re: Incident with Actions and Codespaces

#7

Post by [deleted] » Wed, Oct 05, 2022, 5:05 PM UTC

[deleted]

Re: Incident with Actions and Codespaces

#8

Post by lovehashbrowns » Wed, Oct 05, 2022, 5:47 PM UTC

javierluraschi wrote: post #5

10min troubleshoot time * 10K devs troubleshooting GitHub Actions right now... there goes one year for comunal dev productivity.

I spent a good 20 minutes trying to figure out why actions weren’t running for my PR. Almost lost my marbles until I saw another PR in a different repo also not running actions and I decided to check the GitHub status page. It feels like there should be some sorta warning or alert on GitHub.com when there’s an outage of a component of the site.

Re: Incident with Actions and Codespaces

#9

Post by reidjs » Wed, Oct 05, 2022, 5:53 PM UTC

javierluraschi wrote: post #5

10min troubleshoot time * 10K devs troubleshooting GitHub Actions right now... there goes one year for comunal dev productivity.

Balance that against the time saved by having GH actions instead of managing your own Jenkins server or something and it’s pretty fair

Re: Incident with Actions and Codespaces

#10

Post by lloydatkinson » Wed, Oct 05, 2022, 6:04 PM UTC

fasteddie31003 wrote: post #6

I'm working on a proactive 3rd party downtime detection tool called AwareOps. Here is some of the recent data from Github's incidents https://app.awareops.com/whatisdown/orgs/437dc8b8-2386-4ef5-... .

This is very cool! I had the same idea for a while - I currently just run a .NET cli tool I wrote that reads the various cloud RSS feeds and then goes and scrapes the linked page because the RSS feeds deliberately don’t include the severity etc.

Are you looking for any collaborations or anything? Would be interested in helping if you ever are