Live data from Hacker News

Incident with GitHub Actions, API requests, Codespaces, Git operations, Issues

githubstatus.com

71–80 of 123 posts

Re: Incident with GitHub Actions, API requests, Codespaces, Git operations, Issues

#71
post #51

Earlier quoted context omitted.

It's yellow lights across the board, literally nothing is green. That's usually indicative of some sort of software infrastructure level failure or cascade failure, not an application-level failure, which usually manifests as one or two specific services going down (depending on how you define "infrastructure" and "application" - with IAC, arguably the software defined infrastructure _is_ an application). I doubt its…

Only load balancer (as an infrastructure) can hit the lights across the board. Not much else.

Databases, Caches or the authentication service? For me read-only requests are working fine and I've not seen any issues. Submitting new contents (e.g. comments) is where it's failing for me. It might be that their database primary is falling over.

Re: Incident with GitHub Actions, API requests, Codespaces, Git operations, Issues

#72

Earlier quoted context omitted.

It's been like that for at least 6 hours, randomly appearing. I would take a pause and try again and then it would work, but now it's definitely much more persistent. Guess it's time to go play some video games.... https://xkcd.com/303/

Here you go: $ while ! git push my; do sleep 1; done Works for me eventually, although commits do not appear in web interface (they do in the actual repository).

Having been on the receiving end of things like this: please, make the sleep longer. Adding more requests to already malfunctioning system is not a good way to help in fixing it.

Re: Incident with GitHub Actions, API requests, Codespaces, Git operations, Issues

#74

Whew, glad I decided to scroll HN right now. I've been puzzling over why I'm getting "! [remote rejected] master -> master (Internal Server Error)" as well while trying to push and decided to take a break.

Time to take some coffee and configure Vim

Don't you guys have other features and stuff to work on locally? What is this "time to take a break when GitHub is down"? I'm saying this a bit tongue in cheek btw :)

Re: Incident with GitHub Actions, API requests, Codespaces, Git operations, Issues

#75
post #35

I'm finding that pushes do go through eventually, this is probably grossly irresponsible, so I don't recommend its use, but I remembered I had this old alias to "push harder" in my ~/.gitconfig: [alias] thrust = "!f() { until git push $@; do sleep 0.5; done; }; f" I've done a few pushes so far, and found that it's going through in <10 tries or so.

The delay makes me think you should use the German word for thrust

Re: Incident with GitHub Actions, API requests, Codespaces, Git operations, Issues

#76

Whew, outage timestamps in UTC. Now I won't have to know what time is it California, and if California currently has PST, PDT, PTSD, etc

To anyone who is reading this and genuinely wants to know: it's PDT, UTC-7.

Re: Incident with GitHub Actions, API requests, Codespaces, Git operations, Issues

#77
post #20

These incidents have to hurt Azure's brand value. It's a monster task to run something as big as GitHub, if they ever get it stable it will lend a lot of credibility to Microsoft's cloud skills.

Serious questions: 1) Is GitHub runing under Azure's technology stack? 2) Is GitHub under Azure's mamagement (in contrast to Visual Studio's team)? I'm not sure about two but I'm pretty sure that GitHub doesn't run under Azure at all, considering that GitHub has fully separate networking from MSN's/Azure's (and GitHub's machines do pingback unlike most of Microsoft's machines which don't).

The last time I checked, the only meaningful parts of GitHub that ran on Azure was/is Actions. Everything else is AWS.

Re: Incident with GitHub Actions, API requests, Codespaces, Git operations, Issues

#79

And of course my developer teammates are still trying to merge PRs. I don't care that it works "some of the time"! Don't mess with the repos when the repo host is having seemingly random issues.

For example: while actions are down, branches can be merged without ci tests passing, even for protected branches. This just happened on one of my repos.

Re: Incident with GitHub Actions, API requests, Codespaces, Git operations, Issues

#80
post #11

Here we go again. GitHub going completely down at least once a month as I said. [0] So nothing has changed. That is excluding the smaller intermittent issues. Let's see if anyone implemented a self-hosted backup or failsafe just in case. Oh dear. [0] https://news.ycombinator.com/item?id=30149071

The entire point of git is that it's decentralized, lol. If I've cloned locally like millions of people do daily, I have a backup.

Good point! This would have been a bigger issue back in the days of cvs and svn.
Post reply on HN