Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

GitHub Actions Incident 29.3

githubstatus.com

Post reply on HN
Print view View original

31–40 of 56 posts

Re: GitHub Actions Incident 29.3

#31

Post by _gabe_ »

nimbius wrote: post #25

how is it in just five years microsoft has managed to pedal this once vibrant and bustling community of developers and creatives into a roaring dumpster fire of sketcky GPL breaking copilot AI and endless seemingly random outages. https://www.githubstatus.com/history github has had 55 outages in 3 months. thats nearly an outage every two days. the last six months of 2022 had 74 outages. In many shops thats tangibly w…

> how is it in just five years microsoft has managed to pedal this once vibrant and bustling community of developers and creatives into a roaring dumpster fire of sketcky GPL breaking copilot AI and endless seemingly random outages.

I mean Github Actions was released 5 years ago[0]. I imagine the infrastructure for actions is more susceptible to outages than the fairly simple features Github offered previously. It makes sense that the number of outages would increase with the additional complexity in the infrastructure.

[0]: https://resources.github.com/devops/tools/automation/actions...

Re: GitHub Actions Incident 29.3

#32

Post by riffic »

jacobsenscott wrote: post #28

Nobody's stock goes down when actions go down. Nobody's stock goes up when actions are working. But everyone's stock goes up when you have mass layoffs. Working as designed.

not sure what the SLA is on Actions but outages are a regular occurrence with these kinds of systems and are incredibly expensive to move to the next 9 of availability.

it's certainly a risk you'll need to evaluate when planning your desired build process.

Re: GitHub Actions Incident 29.3

#33

Post by andrewxdiamond »

zamnos wrote: post #27
andrewxdiamond wrote: post #15

Earlier quoted context omitted.

As someone who runs one of the most global APIs in the world, I promise you, I do in fact sleep

What's your pager rotation like? I want to say you have follows-the-sun, and so your on-call shifts are 12-hours long and you swap with a team on the other side of the world from you so you can get said sleep, but I don't want to just assume that.

Dayshifts are 9-5, night shifts are 5-9. Same team rotates through both. I have done plenty of overnight oncalls.

Some teams do follow the sun type rotations, but my team is all in Seattle.

Re: GitHub Actions Incident 29.3

#34

Post by andrewxdiamond »

bastardoperator wrote: post #17
andrewxdiamond wrote: post #15

Earlier quoted context omitted.

As someone who runs one of the most global APIs in the world, I promise you, I do in fact sleep

That's why a global team is important, when you sleep they work, when they sleep you work. Work is constant when you're servicing the globe.

And work is constantly slow as well, since it’s impossible to get people in the room at the same time

Re: GitHub Actions Incident 29.3

#35

Post by dunno7456 »

carlmr wrote: post #23
rvz wrote: post #5

Once again, just two days ago [0], the whole of GitHub went down, after the RSA key leakage and the certificate key expiry on its user facing site. It is also apparent that GitHub Actions has chronically been struggling to operate normally for at least once a month for years. There is no question that GitHub has been more unreliable than if you were to use a self-hosted GitLab or Gittea instance yourself as I said be…

Gittea is great but it doesn't replace GHA, only the rest of GitHub.

Gitea + Woodpecker CI works pretty nicely.

Re: GitHub Actions Incident 29.3

#36

Post by kingds »

nightfly wrote: post #19
tmpz22 wrote: post #18

Earlier quoted context omitted.

A pull request is a process which can merge new code into existing code. "Software Engineer John was tasked to add a new logo to the website, when he was done he submitted a pull request of his feature branch into his organization's github repository for the website so that his team members could approve the changes before automation (like Github Actions ) deployed live as a new version of the website."

Gitlab has "merge requests" the comment you're replying was trying to point out an error in an annoying way

i tried to add a lil laughing emoji to indicate that my comment was in jest but apparently those get stripped out of comments? my bad.

Re: GitHub Actions Incident 29.3

#37

Post by bencevans »

temp_account_32 wrote: post #4

Funnily enough, GitLab is also melting down at the moment, with pipelines not running and pull requests not functioning: https://status.gitlab.com/

Also causing issues due to a change that's made release source tars change [1] (changing hash), so build systems are rejecting [2].

1: https://gitlab.com/gitlab-org/gitlab/-/issues/402616 2: https://github.com/microsoft/vcpkg/issues/30481

Re: GitHub Actions Incident 29.3

#39

Post by bastardoperator »

andrewxdiamond wrote: post #34
bastardoperator wrote: post #17

Earlier quoted context omitted.

That's why a global team is important, when you sleep they work, when they sleep you work. Work is constant when you're servicing the globe.

And work is constantly slow as well, since it’s impossible to get people in the room at the same time

Why does everyone need to be in the room? I have a groomed backlog and can talk to people async as needed. We also record meetings if you missed them and depending on the context of the meeting or importance, we'll hold timezone friendly meetings for everyone as required.

Re: GitHub Actions Incident 29.3

#40

Post by Rapzid »

bencevans wrote: post #37
temp_account_32 wrote: post #4

Funnily enough, GitLab is also melting down at the moment, with pipelines not running and pull requests not functioning: https://status.gitlab.com/

Also causing issues due to a change that's made release source tars change [1] (changing hash), so build systems are rejecting [2]. 1: https://gitlab.com/gitlab-org/gitlab/-/issues/402616 2: https://github.com/microsoft/vcpkg/issues/30481

Holy cow. The cardinal DevOps sin of changing the contents of a versioned file.
Post reply on HN