Incident with GitHub Actions, API requests, Codespaces, Git operations, Issues • Hacker News Board

Re: Incident with GitHub Actions, API requests, Codespaces, Git operations, Issues

#11

Post by rvz » Thu, Mar 17, 2022, 2:18 PM UTC

Here we go again. GitHub going completely down at least once a month as I said. [0] So nothing has changed. That is excluding the smaller intermittent issues. Let's see if anyone implemented a self-hosted backup or failsafe just in case.

Oh dear.

[0] https://news.ycombinator.com/item?id=30149071

Re: Incident with GitHub Actions, API requests, Codespaces, Git operations, Issues

#12

Post by Wavelets » Thu, Mar 17, 2022, 2:21 PM UTC

Whew, glad I decided to scroll HN right now. I've been puzzling over why I'm getting "! [remote rejected] master -> master (Internal Server Error)" as well while trying to push and decided to take a break.

Re: Incident with GitHub Actions, API requests, Codespaces, Git operations, Issues

#13

Post by candiddevmike » Thu, Mar 17, 2022, 2:24 PM UTC

This is causing actions jobs to hang after completing, consuming precious minutes. I don't think I've ever seen a refund when this happens, so I recommend everyone check their jobs and cancel them for now.

Re: Incident with GitHub Actions, API requests, Codespaces, Git operations, Issues

#14

Post by forgingahead » Thu, Mar 17, 2022, 2:24 PM UTC

Wavelets wrote: post #12

Whew, glad I decided to scroll HN right now. I've been puzzling over why I'm getting "! [remote rejected] master -> master (Internal Server Error)" as well while trying to push and decided to take a break.

It's been like that for at least 6 hours, randomly appearing. I would take a pause and try again and then it would work, but now it's definitely much more persistent.

Guess it's time to go play some video games....

https://xkcd.com/303/

Re: Incident with GitHub Actions, API requests, Codespaces, Git operations, Issues

#15

Post by WFHRenaissance » Thu, Mar 17, 2022, 2:24 PM UTC

Looks like the drinking started early at GitHub... good on them!

Re: Incident with GitHub Actions, API requests, Codespaces, Git operations, Issues

#16

Post by PeterBarrett » Thu, Mar 17, 2022, 2:25 PM UTC

One of our systems runs AWS code repository in parallel to Github and builds are triggered from there (but not in us-east-1). Time to migrate the rest of our systems to having that fallback.

Re: Incident with GitHub Actions, API requests, Codespaces, Git operations, Issues

#17

Post by lebski88 » Thu, Mar 17, 2022, 2:27 PM UTC

It's almost the same time as their incident yesterday too. Although today the scope is wider - yesterday it was Webhooks and Actions. Today core git is broken as well as the APIs.

Re: Incident with GitHub Actions, API requests, Codespaces, Git operations, Issues

#18

Post by ahmadrosid » Thu, Mar 17, 2022, 2:27 PM UTC

Wavelets wrote: post #12

Whew, glad I decided to scroll HN right now. I've been puzzling over why I'm getting "! [remote rejected] master -> master (Internal Server Error)" as well while trying to push and decided to take a break.

Same here got rejected when push. ! [remote rejected] HEAD -> main (Internal Server Error)

Re: Incident with GitHub Actions, API requests, Codespaces, Git operations, Issues

#19

Post by alexambarch » Thu, Mar 17, 2022, 2:29 PM UTC

I'm unable to even sign out. It gives me a 500 and then drops me right back at the homepage on a refresh.

Re: Incident with GitHub Actions, API requests, Codespaces, Git operations, Issues

#20

Post by arpinum » Thu, Mar 17, 2022, 2:30 PM UTC

These incidents have to hurt Azure's brand value. It's a monster task to run something as big as GitHub, if they ever get it stable it will lend a lot of credibility to Microsoft's cloud skills.