Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Incident with GitHub Actions

githubstatus.com

Post reply on HN
Print view View original

41–50 of 50 posts

Re: Incident with GitHub Actions

#41

Post by iampims »

rvz wrote: post #24
hypeatei wrote: post #12

I think it's easy to pile-on and say "GitHub is down again! Should've self hosted lol!". When, in reality, it's one service having issues and not the whole site. These incidents also seem to be resolved quickly. Downtime is not the end of the world.

Judging from last month [0] and my own comment chain, [1] it is not so good and it is like as if it is guaranteed to go down each month. > Downtime is not the end of the world. What if you needed to push that critical change and it is down and all you could do is wait? What if you hosted your website on GitHub Pages? Maybe you use GitHub Actions (I assume most do here and are paying for it for their teams). Surely pe…

You always need a break glass solution for these kind of events, SaaS or self-hosted, something will break at an inopportune time and you need to be able to move forward.

Re: Incident with GitHub Actions

#42

Post by redisman »

BeefWellington wrote: post #39
somehnacct3757 wrote: post #34

Earlier quoted context omitted.

We aren't a big enough team. The resources to do such a thing would mean less time focused on customers. Maybe after another doubling we will have these options available. We should be a perfect fit at this stage for a hosted solution. Github is making themselves unviable for a company at this stage (low $XX million ARR)

"We aren't big enough to self-host" is an example of the effectiveness of SaaS sales pitches. Don't get me wrong, there's a lot of good reasons to go cloud. There's not a lot of good reasons for a build system to be completely down with no backup plans when your SaaS provider has an outage. What does even just Github Actions' availability look like for March? 90%? That's pretty brutal for what should be a five nines…

We can run the CI by hand or move it to another platform. But it takes some time and effort. This uptime is abysmal and has already stalled my work days twice in the last few weeks so maybe it’s worth the change

Re: Incident with GitHub Actions

#43

Post by redisman »

grumple wrote: post #30
RapperWhoMadeIt wrote: post #11

I would maybe start migrating repos away from GitHub, since they have proven to be quite unreliable. Nonetheless, I must say I do appreciate GitHub's UI and even their CLI is quite nice. Is there a service out there that basically provides the same good user experience, CI/CD and has no costs for public repos? Apart from the unreliability I can't complain about GitHub.

> Is there a service out there that basically provides the same good user experience, CI/CD and has no costs for public repos? The answer is no. Other UIs are much worse in my experience. I haven't given Gitlab a good shake in a while though. That said, the network effects of Github are pretty significant. Everybody's on it, lots of major repos and orgs use it.

GitHub the source store is fine and very stable. It’s all the new features that seem half baked. Unfortunately we’re using Actions because it was a Faster workflow to get builds out. Not so much right now

Re: Incident with GitHub Actions

#44

Post by somehnacct3757 »

BeefWellington wrote: post #39
somehnacct3757 wrote: post #34

Earlier quoted context omitted.

We aren't a big enough team. The resources to do such a thing would mean less time focused on customers. Maybe after another doubling we will have these options available. We should be a perfect fit at this stage for a hosted solution. Github is making themselves unviable for a company at this stage (low $XX million ARR)

"We aren't big enough to self-host" is an example of the effectiveness of SaaS sales pitches. Don't get me wrong, there's a lot of good reasons to go cloud. There's not a lot of good reasons for a build system to be completely down with no backup plans when your SaaS provider has an outage. What does even just Github Actions' availability look like for March? 90%? That's pretty brutal for what should be a five nines…

We have other tools that are self hosted and they require regular maintenance to security patch, write storage retention policies when some stupid log fills up, have backups, migrate cloud images, stop using plugin with deprecated api, etc. This stuff can also be unplanned and even scheduled becomes a regular chore taking time away from customers.

I know the on-prem sharks smell blood in the water but on-prem is not all upside and not the default answer to hosted services having downtime.

For example we could be on another DevOps SaaS and have zero downtime in March.

Re: Incident with GitHub Actions

#45

Post by jitl »

izietto wrote: post #40
K0nserv wrote: post #27

Earlier quoted context omitted.

Product development has also accelerated since the acquisition. Lots of interesting stuff has been released since it

Flops have been released too (Copilot I'm looking at you)

Copilot is amazingly good. I find it particularly helpful for filling out enum unions or building other kinds of conceptually linked types that don’t have enough semantic linkage for existing autocomplete systems to figure out my intention.

Re: Incident with GitHub Actions

#46

Post by JMS2021 »

rurban wrote: post #25
Shadonototra wrote: post #21

Earlier quoted context omitted.

scaling is a hard problem to solve, so props to them for doing it without too much issues, overall.. but damn, since the microsoft acquisition, they are having more issues more frequently

Actions runs on Azure, actions were only possible with Microsoft.

Action Runners are basically NodeJS applications packaged self-contained. You can run Action Runners on a RPi if you want.

GitHub.com (not GHE) use Action Runners deployed using K8S. This can be done anywhere that supports K8S.

Obviously MS used their own server infra, but to imply GitHub Actions needs Azure is plain false.

Re: Incident with GitHub Actions

#47

Post by aaomidi »

JMS2021 wrote: post #46
rurban wrote: post #25

Earlier quoted context omitted.

Actions runs on Azure, actions were only possible with Microsoft.

Action Runners are basically NodeJS applications packaged self-contained. You can run Action Runners on a RPi if you want. GitHub.com (not GHE) use Action Runners deployed using K8S. This can be done anywhere that supports K8S. Obviously MS used their own server infra, but to imply GitHub Actions needs Azure is plain false.

I mean more so in a way that they're eating that cost. Much harder to do when you're not making huge cash from another segment of your offerings.

Re: Incident with GitHub Actions

#48

Post by matthewmacleod »

izietto wrote: post #40
K0nserv wrote: post #27

Earlier quoted context omitted.

Product development has also accelerated since the acquisition. Lots of interesting stuff has been released since it

Flops have been released too (Copilot I'm looking at you)

Simmer down, it hasn't even been _released_ yet.

Re: Incident with GitHub Actions

#49

Post by mschuster91 »

Kelteseth wrote: post #36
mschuster91 wrote: post #28

Earlier quoted context omitted.

Probably Atlassian was sick of people never upgrading their old installations and getting hacked for it, and people did not upgrade because it is quite a hassle in the first place, not to mention plugins breaking allll the time. Oh and because cloud forces continuous payment whereas prior many customers simply bought a one year license and went on without renewing support.

We're running a selfhosted Gitlab Premium since 2019. The only two times in the last 3 years we had issues with artifacts not being deleted (causing nightly backups to become 500gb, will be fixed in the next version) and some out of date apt certs to run "apt update". Otherwise, I update Gitlab every month without problems.

Gitlab is a breeze to upgrade when using the Docker distribution. Swap the version number in Kubernetes or the systemd unit file (if you're using naked Docker), restart the service, that's it...

Atlassian's docker images are similarly easy to use, but with everything Atlassian you have a veritable ecosystem of plugins of which almost none are open source so you are out of luck if there are incompatibilities.

Re: Incident with GitHub Actions

#50

Post by rurban »

JMS2021 wrote: post #46
rurban wrote: post #25

Earlier quoted context omitted.

Actions runs on Azure, actions were only possible with Microsoft.

Action Runners are basically NodeJS applications packaged self-contained. You can run Action Runners on a RPi if you want. GitHub.com (not GHE) use Action Runners deployed using K8S. This can be done anywhere that supports K8S. Obviously MS used their own server infra, but to imply GitHub Actions needs Azure is plain false.

financially.

it was created to make good free use of azure. before azure GitHub would not be able to afford a free ci infra. someone has to pay all those servers running for free.

Post reply on HN