Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

GitHub Actions Down

news.ycombinator.com

Post reply on HN
Print view View original

31–40 of 54 posts

Re: GitHub Actions Down

#31

Post by zomglings »

In the last week, there have been at least two ~1 hour periods where actions were stuck in the queue. Even posted about this on the GitHub community, but no response. [0]

As unreliable as GitHub Actions are, their convenience factor and price are right.

We take a very simple measure so we don't get fucked by these kinds of incidents: we don't use any actions from the marketplace.

All our GitHub Actions workflows are bash scripts that we wrote (and which often live in our repos at `deploy/deploy.bash`). The secrets necessary to run these scripts are available to the infrastructure team on 1Password.

This makes it easy for us to deploy manually and retroactively reflect that release on GitHub (e.g. through a tag or a release).

[0] https://github.community/t/github-action-stuck-on-starting-w...

Re: GitHub Actions Down

#33

Post by bob1029 »

We only use GH Actions for check builds right now, which are easily disabled as a temporary measure.

Having the ability to build & deploy your software outside the confines of a cloud vendor is essential to survival. When the automation works, its great. When it doesn't, have a manual process that you can follow on a local workstation.

At the end of the day, you can always email the customer a zip file and walk them through installing the update in production. That is, as long as you didn't make your architecture and CI/CD one in the same thing, in which case you probably need to hit the reset button and try again.

Re: GitHub Actions Down

#34

Post by xiwenc »

zomglings wrote: post #31

In the last week, there have been at least two ~1 hour periods where actions were stuck in the queue. Even posted about this on the GitHub community, but no response. [0] As unreliable as GitHub Actions are, their convenience factor and price are right. We take a very simple measure so we don't get fucked by these kinds of incidents: we don't use any actions from the marketplace. All our GitHub Actions workflows are…

This is an interesting approach. I also dislike the current architecture they are pushing. Actions can break any time. Makes pipelines/actions very fragile and as you pointed not portable either.

The portability could be fixed by having a local cli runner that understand action yamls. Would be interesting to explore this.

For a while i have been fantasizing about an universal pipeline language. Like having LLVM with a unified model that can translate into different vendor implementations.

Re: GitHub Actions Down

#35

Post by rjzzleep »

isbvhodnvemrwvn wrote: post #9
Hamuko wrote: post #8

Earlier quoted context omitted.

I was amused by reports when Microsoft was in talks with Discord that one of the reasons why Microsoft wanted to buy Discord was because they wanted Discord on Azure. Like, was the grand customer acquisition strategy for Azure just acquiring the companies and then migrating over?

My wife works as an AWS/Azure consultant, and she mentions that in our area it's much more common for the non-technical management to push Azure than it is for technology to choose it. Sounds quite IBM-ish/Oracle-ish.

I've had a couple of Fortune 500 companies as clients. Microsoft/Azure is usually brought it as a place that is treated like VMs in the cloud. The setup and management is often handled by Accenture and Infosys. Impossible that that decision was made by Engineering. In fact those Accenture managed setups are almost unusable for engineers. I can't even begin to fathom how much these companies spend on Accenture to setup Azure in a fashion that you can't do anything.

The worst part about Azure for me is always the list of undocumented bugs you run into. On the surface it looks like everything started as an AWS equivalent, but when you have to drill down on something it almost always has some weird issues that you then find as unresolved complaints on some MS managed github issue list.

But hey, maybe I was just luckier with the other cloud providers.

Re: GitHub Actions Down

#37

Post by tailspin2019 »

xiwenc wrote: post #34
zomglings wrote: post #31

In the last week, there have been at least two ~1 hour periods where actions were stuck in the queue. Even posted about this on the GitHub community, but no response. [0] As unreliable as GitHub Actions are, their convenience factor and price are right. We take a very simple measure so we don't get fucked by these kinds of incidents: we don't use any actions from the marketplace. All our GitHub Actions workflows are…

This is an interesting approach. I also dislike the current architecture they are pushing. Actions can break any time. Makes pipelines/actions very fragile and as you pointed not portable either. The portability could be fixed by having a local cli runner that understand action yamls. Would be interesting to explore this. For a while i have been fantasizing about an universal pipeline language. Like having LLVM with…

> The portability could be fixed by having a local cli runner that understand action yamls. Would be interesting to explore this.

This is a great idea. I guess the challenge would be keeping up with the more advanced aspects of Actions, eg. spinning up multiple VMs during the build process and other things that have a heavy infrastructure (or platform-specific) element.

I'm currently caught by the GitHub Actions downtime, but like the GP, all my build scripts generally make very light use of the platform-specific features and generally I keep most of the build logic in a build.sh file.

So I can build production builds locally if I need to - but a CLI that lets you "properly" run the GitHub yml files locally would be very interesting.

Re: GitHub Actions Down

#38

Post by ddlsmurf »

dap wrote: post #30
ddlsmurf wrote: post #13

Earlier quoted context omitted.

Personally I'd rather they focus on the issue rather than negotiating with PR and legal to formulate an e-mail. If you rely on an external service and don't monitor it that's on you.

Have you tried monitoring GitHub Actions? It’s not uncommon for me to find that actions just don’t run for some reason. The docs are so incomplete that it’s hard for me to know why.

I can't say github actions but azure devops yes, I have an http endpoint I want stuff to hit with outcome and if it's not I get bugged. Anything is going to break, for external stuff this is the only way to estimate the cost/benefit of a contingency.

Re: GitHub Actions Down

#39

Post by Aeolun »

Sebb767 wrote: post #21
Aeolun wrote: post #18

Earlier quoted context omitted.

The fact that it’s down is not the problem, but it’s painful that there’s nothing I can do to fix it. That’s the part I like about self hosting. It may ultimately be down more often, but I never have to tell someone “Nothing to be done, we wait.”.

Which is better for you, but for your company, "wait, I'm on it" and "wait, they're on it" does not make a world of difference. What's better, on the other hand, is that you can schedule expected and possible downtimes to a time that causes the least impact to your company; with a SaaS, an update might cause you problems any time.

While I sort of agree, I guess that’s that’s arguable. My bosses really like telling theirs that we are doing something about it too.

In these kinds of situations, you often end up in a situation where they say ‘the problem is resolving itself in region x’, where region x is not relevant to you at all. If you are fixing your own setup you can focus on exactly what is most important (to you) first.

Re: GitHub Actions Down

#40

Post by vendiddy »

bob1029 wrote: post #33

We only use GH Actions for check builds right now, which are easily disabled as a temporary measure. Having the ability to build & deploy your software outside the confines of a cloud vendor is essential to survival. When the automation works, its great. When it doesn't, have a manual process that you can follow on a local workstation. At the end of the day, you can always email the customer a zip file and walk them…

Great advice. How is this managed where you are? Do you share the same code between CI and the local workstation?
Post reply on HN