Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

GitHub was down

github.com

Post reply on HN
Print view View original

231–240 of 288 posts

Re: GitHub was down

#231

Post by smsm42 »

temp0826 wrote: post #228
sebmellen wrote: post #7

I've never seen an outage this big. Even the homepage doesn't load. We've had recurrent issues with Actions not running, but this seems a lot bigger. The status page says all is well, though: https://www.githubstatus.com/ . Hilarious.

Used to work ops at AWS. I don't know if it's still the case but it required VERY HIGH management approval to actually flip any lights on their "status page" (likely it was referenced in some way for SLAs and refunding customers).

That is an excellent illustration to Goodhart's law. We're going to have this avesome status page, but since if we update it the clients would notice the system is down, we're going to put a lot of barriers to putting the actual status on that page.

Also probably a class action suit lurking somewhere in there eventually.

Re: GitHub was down

#232

Post by rvz »

mrala wrote: post #208
rvz wrote: post #92

Earlier quoted context omitted.

I guess when GitHub goes down, it is somehow strangely tolerated for years even after the acquisition and goes down more times than Twitter. When the latter encounters a speed-bump, just like the 'interview' with Trump, it's global news because a Mr Elon Musk owns it. Both seem to be doing too much all at once. But really it is worse with Github if this is what Microsoft stewardship is incidents every, single week an…

What makes you say it’s “somehow strangely tolerated” when GitHub goes down? What’s the point of bringing up twitter? It is strange to seek victimhood for a petulant billionaire. Of course, it is worse with GitHub because GitHub actually provides useful functionality.

> What makes you say it’s “somehow strangely tolerated” when GitHub goes down?

The same folks complaining about something at GitHub going down are the same people that stay and are willing to tolerate the regular incidents and chaos on the site.

It is the fact that not only the Github incidents have been happening for years, it has gotten worse as there is an incident every month.

> Of course, it is worse with GitHub because GitHub actually provides useful functionality.

That isn’t an excuse for tolerating regular downtime for a site with over 100m+ active users, especially with it running under Microsoft stewardship who should know better.

Any other site with that many users and with a horrendous record of downtime like Github would be rightfully branded as unreliable. No excuses.

Re: GitHub was down

#233

Post by __float »

croemer wrote: post #127
dghlsakjg wrote: post #104

https://www.githubstatus.com/ This is a pretty good place to check. The lag is pretty minimal traditionally. At the time of posting everything is broken.

Not sure what you mean by minimal lag. The status page showed all green for at least 10min while everyone got unicorns.

I got unicorns on the status page, even.

Re: GitHub was down

#234

Post by bigiain »

ergocoder wrote: post #27
sebmellen wrote: post #7

I've never seen an outage this big. Even the homepage doesn't load. We've had recurrent issues with Actions not running, but this seems a lot bigger. The status page says all is well, though: https://www.githubstatus.com/ . Hilarious.

I wonder why the status just doesn't ping github.com for 200. That seems easy to do.

To be fair - I really couldn't care less is the homepage is loading or not.

So long as I can fetch/commit to my repos, pretty much everything else is of secondary, tertiary, or no real importance to me.

(At work, I do indeed have systems running that monitor 200 statuses from client project homepages, almost all of which show better that 99.999% uptimes. And are practically useless. Most of them also monitor "canary" API requests which I strive to keep at 99.99% but don't always manage to achieve 99.9% - which is the very best and most expensive SLA we'll commit to.)

Re: GitHub was down

#236

Post by __turbobrew__ »

weystrom wrote: post #99

A reminder of how centralized and dependent the whole industry has become on GH, which is ironic, considering that git itself is designed to be decentralized. Good opportunity to think about mirroring your repos somewhere else like Gitea or Gitlab.

I’m kind of surprised that gitlab doesn’t have a larger market share given that you can run it air gapped on-prem without too much fuss.

Re: GitHub was down

#237

Post by __float »

outworlder wrote: post #225
curtis3389 wrote: post #97

Earlier quoted context omitted.

I recently looked into using Git in a decentralized way. It's actually pretty easy! When you would usually create a PR, you use `git format-patch` to create a patch file and send that to whoever is going to merge it. They create a branch and use `git am` to apply the patch to it, review the changes, and merge it to main. It is nice that git supports multiple remotes, though. It feels good to know that `git push` migh…

I used to work at a company with very draconian policies. Whenever I needed to update some code on a public GitHub repository, I would just push to a remote that was a flash drive. Plug it in my machine at home, pull from that remote, push to origin. I also had to setup a bidirectional mirror back when bandwidth to some countries was restricted. We would push and pull as normal, and a job would keep our mainline in s…

Draconian policies...but not security ones? Why were USB drives not blocked?

Re: GitHub was down

#238

Post by wrs »

colimbarna wrote: post #218
cbates wrote: post #102

Earlier quoted context omitted.

Seems slightly unproffesional for a massive company like Github/Microsoft.

It's closer to the truth than you usually get. They're having a bad day, it's completely true. It's the start of my day, but I guess this is the middle of the night for them. There's no such thing as unicorns, but that just highlights the metaphorical nature of the remaining claim - getting Unicorns under control means solving their problems. Normally "professional" corporate speak means avoiding saying anything whos…

GitHub is (was?) a Rails application, so it was probably originally running behind Unicorn [0], if it isn’t still. So the unicorns are (were) real.

[0] https://en.wikipedia.org/wiki/Unicorn_(web_server)

Re: GitHub was down

#239

Post by fragmede »

bamboozled wrote: post #202
kgrax01 wrote: post #122

Could it have been brought down intentionally? Related to this? https://www.bleepingcomputer.com/news/security/github-action...

How would customer credentials being leaked be part of an outage of this size ?

Because there are worse things than being down; if the front page got hacked and is spewing gore or CSAM or PII or creds, for example.

Re: GitHub was down

#240

Post by owyn »

tabbott wrote: post #10
twp wrote: post #5

https://www.githubstatus.com/ reports no problems, but it's clearly down for a lot of people (including me).

It is kinda amazing how consistently status pages show everything fine during a total outage. It's not that hard to connect a status page to end-to-end monitoring statistics...

Once in the past I did actually have an incident where the site went down so hard that the tool that we used to update the status page didn't work. We did move it to a totally external and independent service after that. The first service we used was more flaky than our actual site was, so it kept showing the site down when it wasn't. So then we moved to another one, etc. Job security. :)
Post reply on HN