Status.github.com: “We're failing over a data storage system” • Hacker News Board

Re: Status.github.com: “We're failing over a data storage system”

#81

Post by ajb257 » Mon, Oct 22, 2018, 6:58 AM UTC

keyle wrote: post #20

I don't know if they keep changing the text updates with a slightly different version to - prove it's a human that typed it - there is code the prevents repeating twice the same message either way it's entertaining... But it's Monday morning in Australia and we need to release! (yep we do this via pr/tagging etc.)

Given the latest update is "We are currently in the later stages of a restore operation, with the aim of serving fully consistent data within the next 2 hours", I can only assume someone at GH read your comment!

Re: Status.github.com: “We're failing over a data storage system”

#82

Post by olingern » Mon, Oct 22, 2018, 6:58 AM UTC

jypepin wrote: post #67

It's been over 6hrs without update, except the hourly message which states the same. I'm really looking forward this post mortem, but having been in the situation where I've had to deal with large scale outages like this one, I guarantee some engineers are having a bad time right now, and I feel for them. Github engineers, if you are reading me (probably not), KEEP IT UP, it happens to the best of us! <3

Posted at 15:51 Japan Standard Time

"...with the aim of serving fully consistent data within the next 2 hours."

That's somewhat significant

Re: Status.github.com: “We're failing over a data storage system”

#83

Post by [deleted] » Mon, Oct 22, 2018, 6:58 AM UTC

reindeerer wrote: post #65

theSage wrote: post #45

I'm really looking forward to the post mortem that comes out of this (if it does). I always learn a lot from reading those.

A post-mortem from Microsoft ? nfw

[deleted]

Re: Status.github.com: “We're failing over a data storage system”

#84

Post by nodesocket » Mon, Oct 22, 2018, 6:59 AM UTC

I created a gist four hours ago, which as of now is not returning and showing a 404. Hopefully the gist did not get lost.

Re: Status.github.com: “We're failing over a data storage system”

#85

Post by samontar » Mon, Oct 22, 2018, 7:07 AM UTC

steventhedev wrote: post #54

guywhocodes wrote: post #51

What's strange to me is how many hours we've been given the same message. Today is an important day for my organization and not getting any information that hints how long this will last is a huge problem for our planning.

Are you a paying customer of GitHub Enterprise? If not, then you're getting your money's worth. Snark aside, this is a great time to reassess your deployment strategies and look into things like local apt and pypi proxies. I'm confident you can find similar projects that will transparently cache your dependencies.

If you’re paying for GHE you don’t have any trouble. That’s delivered as an appliance you host.

Re: Status.github.com: “We're failing over a data storage system”

#86

Post by josteink » Mon, Oct 22, 2018, 7:07 AM UTC

Seems like they're on their way back up:

> We are currently in the later stages of a restore operation, with the aim of serving fully consistent data within the next 2 hours."

Re: Status.github.com: “We're failing over a data storage system”

#87

Post by _shadi » Mon, Oct 22, 2018, 7:08 AM UTC

I always thought it is silly that we pay for both private repos on github and host our own enterprise github on premise, turns out whoever have set this up knew what they were doing.

Re: Status.github.com: “We're failing over a data storage system”

#88

Post by samontar » Mon, Oct 22, 2018, 7:09 AM UTC

karaokeyoga wrote: post #4

Is it just me or is that status message wonky? Failing over something to restore access?

It’s just a variant referencing https://en.m.wikipedia.org/wiki/Failover

Re: Status.github.com: “We're failing over a data storage system”

#89

Post by calmconviction » Mon, Oct 22, 2018, 7:11 AM UTC

They probably moved their storage backends to Windows 10

Re: Status.github.com: “We're failing over a data storage system”

#90

Post by perlgeek » Mon, Oct 22, 2018, 7:36 AM UTC

samuel1604 wrote: post #62

ComputerGuru wrote: post #39

I can't add comments to pull requests("you can't do that right now") and any commits pushed to branches are not updating updating the visible status in the web interface, no are newly created branches showing up. However, if you navigate to a new commit directly with its SHA (so you can share it with someone if you really want), it'll show up (so they're just not being indexed). EDIT: Obligatory "that's what happens…

yes, but does it scale tho ? is it DR ? I mean sure fine it's easy and pretty but there is some operation challenge to figure out,

If you host just your own Open Source projects, does it need to scale?