I was waiting entire day for gh actions to work, today everything works as usual
GitHub Actions and Pages are experiencing degraded availability
401–410 of 446 posts
Re: GitHub Actions and Pages are experiencing degraded availability
#402Given the recent history of outages, I wonder how many are seriously considering moving off GitHub for anything other than code hosting. I mean actions / workflows.
To offer a data point (not a tribalist argument): GitHub Actions being down this often is still less expensive than the cost of switching everything somewhere else. Would rather have my team walk away from the PRs and do other tasks than have engineering effort, meetings, design pages, scheduling, etc. for switching over. This is annoying and I'm here because it's down. But it would have to be far worse to come close…
In my experience moving from GH to Forgejo was trivial, largely because the actions are mostly compatible and importing the repos can be automated. The amount of time it took was definitely less than the 5-8 hours of downtime that GHA had yesterday.
Re: GitHub Actions and Pages are experiencing degraded availability
#403Earlier quoted context omitted.
This was posted elsewhere in the discussion: https://damrnelson.github.io/github-historical-uptime/ Went downhill long before LLMs
The missing line on that graph is November 2019 when Github Actions were introduced. It would also show the point where things went wrong for uptime
Before Github Actions places I worked at pushed monliths to a dedicated CI, Github just hosted the repo, now we push microservices to a monolithic deploy pipeline. I'm sure someday we'll get it right, or maybe the robots will for us.
Re: GitHub Actions and Pages are experiencing degraded availability
#404Github CI being down is catastrophically bad.
Re: GitHub Actions and Pages are experiencing degraded availability
#405Re: GitHub Actions and Pages are experiencing degraded availability
#406I don't think this portents anything great for software in general. We're a good year+ into the use LLMs for all major bits of software that we all rely upon and GitHub here is down to one 9 of uptime. I've been using GitHub for a _long_ time, my first commits there go back to August 2009!, and I honestly don't recall GitHub going down as much as it has in the last year. I'm sure there's other things happening in the…
Are these outages caused by introduced bugs, though, or by load issues? As someone who has spent many years working in high load environments, this is not an uncommon pattern. You design a system and it works great. It can handle failures, load spikes, it is horizontally scalable, things are great. You think you figured it out. And then load keeps increasing and you suddenly hit a tipping point where everything keeps…
I agree, no horizontally-scalable system is infinitely scalable in any dimension. You will hit some limit - for example, you might be able to support arbitrary scale in some dimension, but there is a bound on the sustainable rate of change, or the second-order rate of change, and you hit that limit.
You seem to be suggesting though that hitting those limits came as a surprise, and/or there was no future architecture plan to address that when it happens, which I think is the real problem here. You would expect a team with the maturity of Github to understand what those limits are, and plan for them, based on predictable growth in demand.
Re: GitHub Actions and Pages are experiencing degraded availability
#407Earlier quoted context omitted.
I don't think so. GitHub was bought by Microsoft 8 years ago and people have only started complaining about its uptime in the last year or so - exactly correlating with the surge in LLM use.
I don't think so. I've seen those complaints for more than a year. I have an Ops background and I strongly suspect they were given a stupid timeline for the Azure migration. I've got to believe Microsoft have decent Ops people but the management wanted to move faster than was reasonable and screwed it up. Move one thing at a time and double check it all works and you can do a migration like this.
Even on a large timeline there is just a lot of risk when migrating legacy infra to an entirely new system, especially if the original arch grew "organically" and has tones of edge cases.
I always figured these problems were directly related to the migration.
Re: GitHub Actions and Pages are experiencing degraded availability
#408I don't think this portents anything great for software in general. We're a good year+ into the use LLMs for all major bits of software that we all rely upon and GitHub here is down to one 9 of uptime. I've been using GitHub for a _long_ time, my first commits there go back to August 2009!, and I honestly don't recall GitHub going down as much as it has in the last year. I'm sure there's other things happening in the…
Now GITHUB gets to experience debugging the GHA in prod. From my experience, good luck.
Re: GitHub Actions and Pages are experiencing degraded availability
#409What is amazing is people do not switch away from GitHub.
Maybe people are actually pretty tolerant of downtime if other things about the product or service are done right
Re: GitHub Actions and Pages are experiencing degraded availability
#410Earlier quoted context omitted.
It pleases shareholders more to hear about exponential Cloud offering adoption than SLA availability for a developer platform.
Except, lots of devs use github, and see what using Azure means. Management is seeing that downtime too. So what insane person will ever touch Azure willingly? Hotmail used to run on bsd, and they tried to move it to WinNT, back in the day. Same thing. Disaster. And it was held up as a reason to never use winnt for serious work, for years. Microsoft just always shoots themselves in the foot.
I think my age group (~40) is the tail end of widespread dogmatic trust in Microsoft. Once we retire I think they are going to be in serious trouble.