Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

GitHub was down

github.com

Post reply on HN
Print view View original

271–280 of 288 posts

Re: GitHub was down

#271

Post by chrisallenlane »

saul-paterson wrote: post #260
sebmellen wrote: post #7

I've never seen an outage this big. Even the homepage doesn't load. We've had recurrent issues with Actions not running, but this seems a lot bigger. The status page says all is well, though: https://www.githubstatus.com/ . Hilarious.

FWIW, our self-hosted Gitea instance has not had a single second of unplanned downtime in five years we've been running it. And there wasn't much _planned_ downtime because it's really easy to upgrade (pull a new image and recreate the container — takes out the instance for maybe 15 seconds late at night), and full backups are handled live thanks to zfs. Migration to a new host takes another 15 seconds thanks to both…

I've been running Gitea on my homelab for a few months now. It's fantastic. It's like a snapshot of a point in time when GitHub was actually good, before it got enshittified by all of the social and AI nonsense.

I've been moving most of my projects off of GitHub and into Gitea, and will continue to do so.

Re: GitHub was down

#272

Post by TacticalCoder »

sebmellen wrote: post #7

I've never seen an outage this big. Even the homepage doesn't load. We've had recurrent issues with Actions not running, but this seems a lot bigger. The status page says all is well, though: https://www.githubstatus.com/ . Hilarious.

> I've never seen an outage this big.

I remember a time when systems would boast about their "five nines" uptime. It was before anything "cloud" appeared.

Re: GitHub was down

#273

Post by darkangelstorm »

Services that explicitly needed the API were also down, and it wasn't pretty. For example: Minecraft Mod packs that rely on SerializationIsBad all went kerplunk! I'm sure a lot of people were scratching their heads yesterday wondering why they couldn't do anything for a time.

What made me laugh though was when the "X is functioning normally" immediately followed by "X is degraded, continuing to monitor" messages that kept popping up then right back to "normal" again, all in the same 30 second timespan... made me giggle

Re: GitHub was down

#274

Post by Gormo »

curtis3389 wrote: post #97
dlahoda wrote: post #33

I see more and more people use less Github, but some other git solutions. I am afraid to think what to do when GitHub is down for hours (need to learn maillists?). Another reason is that MS may be in phase when it will ask to pay for using GitHub just for reads (rate limiter).

I recently looked into using Git in a decentralized way. It's actually pretty easy! When you would usually create a PR, you use `git format-patch` to create a patch file and send that to whoever is going to merge it. They create a branch and use `git am` to apply the patch to it, review the changes, and merge it to main. It is nice that git supports multiple remotes, though. It feels good to know that `git push` migh…

> I recently looked into using Git in a decentralized way. It's actually pretty easy!

Well, that's how it was designed to work! The whole point of Git is that it's a distributed version control system, and doesn't need to rely on a centralized source of truth.

Re: GitHub was down

#276

Post by consteval »

ParetoOptimal wrote: post #195
xuancanh wrote: post #126

Earlier quoted context omitted.

It's because of the way most companies build their status dashboards. There are usually at least 2 dashboards, one internal dashboard and one external dashboard. The internal dashboard is the actual monitoring dashboard, where it will be hooked up with other monitoring data sources. The external status dashboard is just for customer communication. Only after the outage/degradation is confirmed internally, then the ex…

> The external status dashboard is just for customer communication. Only after the outage/degradation is confirmed internally, then the external dashboard will be updated to avoid flaky monitors and alerts. It will also affect SLAs so it needs multiple levels of approval to change the status, that's why there are some delays. This defeats the purpose of a status dashboard and is effectively useless in practice most o…

From a business perspective, I think given the choice to lie a little bit or be brutally honest with your customers, lying a bit is almost always the correct choice.

Re: GitHub was down

#277

Post by insane_dreamer »

ParetoOptimal wrote: post #195
xuancanh wrote: post #126

Earlier quoted context omitted.

It's because of the way most companies build their status dashboards. There are usually at least 2 dashboards, one internal dashboard and one external dashboard. The internal dashboard is the actual monitoring dashboard, where it will be hooked up with other monitoring data sources. The external status dashboard is just for customer communication. Only after the outage/degradation is confirmed internally, then the ex…

> The external status dashboard is just for customer communication. Only after the outage/degradation is confirmed internally, then the external dashboard will be updated to avoid flaky monitors and alerts. It will also affect SLAs so it needs multiple levels of approval to change the status, that's why there are some delays. This defeats the purpose of a status dashboard and is effectively useless in practice most o…

Liability is their primary concern

Re: GitHub was down

#278

Post by 2YwaZHXV »

bill_lumbergh wrote: post #246
2YwaZHXV wrote: post #226

Earlier quoted context omitted.

makes sense, thanks. the images on the page are all just base64 encoded right into the html

https://github.blog/news-insights/the-library/unicorn/ Unicorn has a slightly different architecture. Instead of the nginx => haproxy => mongrel cluster setup you end up with something like: nginx => shared socket => unicorn worker pools When the Unicorn master starts, it loads our app into memory. As soon as it’s ready to serve requests it forks 16 workers. Those workers then select() on the socket, only serving req…

amazing, thanks!

Re: GitHub was down

#279

Post by Joe_Cool »

colimbarna wrote: post #241
Joe_Cool wrote: post #217

Earlier quoted context omitted.

We installed a private GitLab instance on our own servers exactly out of fear that Github might suddenly alter the deal or just cease operations. Pretty happy with our decision so far.

Do you mean you switched to self-managed GitLab, or you have a self-managed GitLab that you keep around as a backup plan?

Actually both. Our internal closed source projects are only in our GitLab. The open-source stuff is both on GitHub and our GitLab. Since our GitLab instance isn't public we only use the issue tracker on GitHub for public stuff.

Another bonus is that we don't pay Microsoft.

Re: GitHub was down

#280

Post by ParetoOptimal »

consteval wrote: post #276
ParetoOptimal wrote: post #195

Earlier quoted context omitted.

> The external status dashboard is just for customer communication. Only after the outage/degradation is confirmed internally, then the external dashboard will be updated to avoid flaky monitors and alerts. It will also affect SLAs so it needs multiple levels of approval to change the status, that's why there are some delays. This defeats the purpose of a status dashboard and is effectively useless in practice most o…

From a business perspective, I think given the choice to lie a little bit or be brutally honest with your customers, lying a bit is almost always the correct choice.

My ideal would be if regulations which made it necessary that downtime metrics had to be reported with at most somewhere between a 10m and 30m delay as "suspected reliability issue".

If your reliability metrics have lots of false positives, that's on you and you'll have to write down some reason why those false positives exist every time.

Then that company could decide for itself whether to update manually with "not a reliability issue because X".

This lets consumers avoid being gaslighted and businesses don't technically have to call it downtime.

Post reply on HN