It would be wild if they dropped below the "two 9's" metric. I think they would need an additional ~16hr of outage in the 90 day rolling period.
GitHub is going for “eight 8’s” at this rate.
Incident with multple GitHub services
81–90 of 136 posts
Re: Incident with multple GitHub services
#82There was another really bad incident today: https://www.githubstatus.com/incidents/zsg1lk7w13cf
> We have resolved a regression present when using merge queue with either squash merges or rebases. If you use merge queue in this configuration, some pull requests may have been merged incorrectly between 2026-04-23 16:05-20:43 UTC.
We had ~8 commits get entirely reverted on our default branch during this time. I've never seen a github incident quite this bad.
Re: Incident with multple GitHub services
#83Earlier quoted context omitted.
I've tried for 15 years to have my homelab, but always get lost in the complexity after a year or so, in the past. About 3 years ago I gave NixOS a try instead for managing everything, which suddenly made everything easier (counter-intuitively perhaps) as now I can come back after months and still understand where everything is and how it works after just reading. Setting up Forgejo + runners declaratively is probabl…
Just to echo what others are saying: NixOS and Proxmox are the answer. I run both right now, but I am in the process of just running NixOS on everything. NixOS really is that good, particularly for homelabs. The module system and ability to share them across machines is really a superpower. You end up having a base config that all machines extend essentially. Same idea applies to users and groups. One of the other bi…
I mean this in a good way, but I'm slightly chuckling to myself that it reads like people are just discovering IaC...on HN. That's all Nix configs are, at the end of the day.
No matter the tool, manage your environment in code, your life becomes much easier. People start and then get addicted to the ClickOps for the initial hit and then end up in a packed closet with a one way ticket to Narnia.
This happens in large environments too, so not at all just a home lab thing.
Re: Incident with multple GitHub services
#84Earlier quoted context omitted.
The idea of a homelab is appealing to me, but then I actually start building one and get tired of it quickly. When I’ve been fixing broken systems at work all day I don’t really want to have to be my own sysadmin too. I’ve got a nice and powerful Minisforum on my desk that I bought at Christmas not even switched on.
I've tried for 15 years to have my homelab, but always get lost in the complexity after a year or so, in the past. About 3 years ago I gave NixOS a try instead for managing everything, which suddenly made everything easier (counter-intuitively perhaps) as now I can come back after months and still understand where everything is and how it works after just reading. Setting up Forgejo + runners declaratively is probabl…
Maybe my needs are simpler. But I just made do with systemd services and apt (debian). I've also setup Incus for the occasional software testing and playing around. After using OpenBSD as a daily driver, I'm more keen with creating a native package for the OS/Distro than wrangling docker compose files.
Re: Incident with multple GitHub services
#85I've been on a somewhat binge to move a bunch of stuff to self-hosting at home. Yesterday I finally completed my self-hosted Forgejo instance at home, together with Linux, Windows (via VM) and macOS (via Mac Mini) runners/workers for CI/CD, so everything finally lives in-house (literally), instead of all source code + Actions being on GitHub but the infrastructure actually living locally. This is probably the first t…
And once you start self-hosting, you realise how slow the 'modern' web actually is.
I host forgejo on a single NUC with a bunch of other stuff in Proxmox, the page loads in 6ms! Immich is not quite as fast but still a ton faster than Google photos.
Re: Incident with multple GitHub services
#86[flagged]
Re: Incident with multple GitHub services
#87Is Github losing any significant business from all these outages?
Curious because for a long time we as an industry maintained that reliability and brand value are business critical; but seems like they are cared very little now a days.
Happy to be corrected about my perception too.
Re: Incident with multple GitHub services
#88Earlier quoted context omitted.
Just to echo what others are saying: NixOS and Proxmox are the answer. I run both right now, but I am in the process of just running NixOS on everything. NixOS really is that good, particularly for homelabs. The module system and ability to share them across machines is really a superpower. You end up having a base config that all machines extend essentially. Same idea applies to users and groups. One of the other bi…
I mean this in a good way, but I'm slightly chuckling to myself that it reads like people are just discovering IaC...on HN. That's all Nix configs are, at the end of the day. No matter the tool, manage your environment in code, your life becomes much easier. People start and then get addicted to the ClickOps for the initial hit and then end up in a packed closet with a one way ticket to Narnia. This happens in large…
I and many other NixOS users know what IaC is :)
A NixOS config is a bit different because it’s lower level and is configuring the OS through a first-party interface. It is more like extending the distro itself as opposed to configuring an existing distro after the fact.
The other big difference is that it is purely declarative vs. a simulation of a declarative config a la Ansible and other tools. Again, because the distro is config aware at all levels, starting from early boot.
The last difference is atomicity. You can (in theory) rely on an all or nothing config switch as well as the ability to rollback at any time (even at boot).
On top of all this are the niceties enabled by Nix and nixpkgs. Shared binary caches, run a config on a VM, bake a live ISO or cloud VM image from a config (Packer style), the NixOS test framework, etc.
Re: Incident with multple GitHub services
#89Earlier quoted context omitted.
I've tried for 15 years to have my homelab, but always get lost in the complexity after a year or so, in the past. About 3 years ago I gave NixOS a try instead for managing everything, which suddenly made everything easier (counter-intuitively perhaps) as now I can come back after months and still understand where everything is and how it works after just reading. Setting up Forgejo + runners declaratively is probabl…
Maybe my needs are simpler. But I just made do with systemd services and apt (debian). I've also setup Incus for the occasional software testing and playing around. After using OpenBSD as a daily driver, I'm more keen with creating a native package for the OS/Distro than wrangling docker compose files.
Yea, it's always weird to see people say "I'm simplifying my life and reducing my cloud dependencies by running a homelab and self-hosting!" and then they list the dozens of alphabet soup software they're running on it that they're now relying/depending on. "Oh I run 20 VMs and containers and Docker orchestration and Nextcloud and Syncthing and Jellyfin and Plex and Forgejo and Komodo and Home Assistant and Immich and Trilium and Audiobookshelf and another Nextcloud and This Stack and That Pipeline" and oh my god haven't you really just made your computing even worse?
My "homelab" is basically Linux + NFS, with standard development tools.
Re: Incident with multple GitHub services
#90I am this > < close to just running Gogs or Forgejo on some Hetzner boxes, quit my job, charge people for access. Why aren't there like 10 startups doing this yet? Please? I want to give you my money. Just give me a git host that doesn't suck. (All the current ones suck)
Codeberg and Sourcehut are doing it for free, for open source. Corporate probably won't ever move off Github, because they need the prestige of using Github - the actual service quality is completely irrelevant. This is an aspect of the enshittocene epoch - I repeat, quality is irrelevant to corporates.
Corporate will move off GitHub as soon as it loses prestige and they will move on to the next thing.