GitHub must be migrating to Azure...
Actually, they just announced to stay with AWS, as long as things run well. Maybe now they have a reason to.
Status.github.com: “We're failing over a data storage system”
131–140 of 173 posts
Re: Status.github.com: “We're failing over a data storage system”
#132Earlier quoted context omitted.
It is to ensure that the updates get mirrored onto twitter, where exact duplicates can't be posted.
If you include a unique hashtag, would the message be considered unique? In that case, you could include the epoch timestamp as a hashtag :-)
Or even just prefix the tweet with the update ordinal.
Re: Status.github.com: “We're failing over a data storage system”
#133Earlier quoted context omitted.
What did the server cost? How many hours did you spend on it? What's your SLA? Who will be woken up at night when there's an outage? What's your backup and recovery procedure? I mean kudos to you for setting it up but it's a bit naive to believe it's better than a hosted solution right away.
Fair questions. We're a company making WISP/ISP software here in India and our cloud (hosted) offering is also hosted on servers at our office itself[1]. Our infra manages hundreds of thousands of internet subscribers in India. We're a small tech company with 25-30 people or so, with linux kernel experts who have experience of running software on questionable hardware so that service providers can take internet to ne…
What do you make out of curiosity? I run an ISP in the UK.
Re: Status.github.com: “We're failing over a data storage system”
#134Just a few weeks ago my organization was in the position of choosing a version control platform for our repos. I'm so glad we went ahead with self hosted gitlab. We installed it on a CentOS server at our premise, SSL'd via Let's encrypt, I've even set up a dedicated gitlab runner to use Gitlab CI for continuous delivery and so far the testing is progressing pretty smoothly. All this for $0. Update: I agree, gitlab an…
What did the server cost? How many hours did you spend on it? What's your SLA? Who will be woken up at night when there's an outage? What's your backup and recovery procedure? I mean kudos to you for setting it up but it's a bit naive to believe it's better than a hosted solution right away.
Git is tool for developers. If developer can't fix broken server, he's doing something wrong. No need to outsource trivial tasks.
Re: Status.github.com: “We're failing over a data storage system”
#135Earlier quoted context omitted.
Are you a paying customer of GitHub Enterprise? If not, then you're getting your money's worth. Snark aside, this is a great time to reassess your deployment strategies and look into things like local apt and pypi proxies. I'm confident you can find similar projects that will transparently cache your dependencies.
If you’re paying for GHE you don’t have any trouble. That’s delivered as an appliance you host.
You are quite correct. I had meant the paid hosted version. Maybe they used to call it GHE at some point? Dunno why it got stuck in my mind like that...
Re: Status.github.com: “We're failing over a data storage system”
#136Earlier quoted context omitted.
What did the server cost? How many hours did you spend on it? What's your SLA? Who will be woken up at night when there's an outage? What's your backup and recovery procedure? I mean kudos to you for setting it up but it's a bit naive to believe it's better than a hosted solution right away.
Git is tool for developers. If developer can't fix broken server, he's doing something wrong. No need to outsource trivial tasks.
Time spent fixing broken servers is time not spent working on other things. In some cases it makes sense to spend your time that way, in other cases it does not.
Re: Status.github.com: “We're failing over a data storage system”
#137My guess at what happened:
They had a split brain when multiple masters were running. Then they were not able to choose a master to keep because the data in both masters was 'corrupted' so they are now restoring from a backup.
So how do they get data corruption from multiple masters running:
1) Performing reads from slaves during an update operation. If you perform a read from a slave then you might get data from the other master. If you update data on one master based on data from another master then you get data corruption. Probably they don't do this much because if you had any slave lag then you would notice this problem during normal operation. However, they might do it for checking permissions. You can imagine because this data almost never changes it would never show up as a problem normally.
2) Data stored outside of the database. This would be the repositories themselves and cookie-session storage. Imagine repos have an incrementing id. Then if you have two masters a repo gets created with the same id on both masters. This is very bad because now two people can see each others data. You have the same problem with cookies. Imagine if you have user id as an incrementing id. The two masters create a user with the same id and an encrypted cookie or another storage system (redis) stores the user id. Now depending on which master you get routed to you appear as a different user.
Weirdly enough this would always be a problem based on how their failover system works. The only safe way i know how to turn a master-slave system into a safe HA system is how joyent does it (https://github.com/joyent/manatee/blob/master/docs/user-guid... : you basically need a vote at commit time. if you have this then only 1 master can commit). However, I'm guessing most of the time they only have two masters running for EDIT: oh. they said it affected issues & pull requests [they have different dbs for different stuff] so repo and authentication stuff wouldn't apply. oh, they said no data was lost as well so presumably that would exclude a split brain running for a long period of time.
Re: Status.github.com: “We're failing over a data storage system”
#138Just a few weeks ago my organization was in the position of choosing a version control platform for our repos. I'm so glad we went ahead with self hosted gitlab. We installed it on a CentOS server at our premise, SSL'd via Let's encrypt, I've even set up a dedicated gitlab runner to use Gitlab CI for continuous delivery and so far the testing is progressing pretty smoothly. All this for $0. Update: I agree, gitlab an…
Hosted gitlab has more features than github. But you are on the hook for security, config and uptime.
However, it might seem a wise choice given gitlab's history of breaking every week and a half. (However that has improved significantly in recent months.)
Gitlab is a far better tool for enterprise, as it can group repos into projects and have a shared issue tracker at a project level (as well as repo). It also has first class CI integration(git lab runners). All of this beats paying for github[1]
[1] I worked for a very large financial news company that has partially migrated from stash/bitbucket to github. With a mix of public/private repos its a total mess. The tools for managing more than 40 repos simply don't exist. Not only is it more expensive, it doesn't integrate will with Jira, (well thats not entirely github's fault) it literally is just a git HTTP web interface, with a ticketing system bolted on.
Re: Status.github.com: “We're failing over a data storage system”
#139An update has been posted there: https://blog.github.com/2018-10-21-october21-incident-report... > At 10:52 pm Sunday UTC, multiple services on GitHub.com were affected by a network partition and subsequent database failure resulting in inconsistent information being presented on our website. Out of an abundance of caution we have taken steps to ensure the integrity of your data, including pausing webhook events and…
1.5 hour later (a couple minutes ago), despite an estimated time of less than an hour:
> We continue to monitor restores which are taking longer than anticipated. We estimate they will be caught up in an hour and a half.
Re: Status.github.com: “We're failing over a data storage system”
#140Just a few weeks ago my organization was in the position of choosing a version control platform for our repos. I'm so glad we went ahead with self hosted gitlab. We installed it on a CentOS server at our premise, SSL'd via Let's encrypt, I've even set up a dedicated gitlab runner to use Gitlab CI for continuous delivery and so far the testing is progressing pretty smoothly. All this for $0. Update: I agree, gitlab an…
> makes us feel like we're more in control and are somehow saving some money This is precisely the problem with developers. You decided to go with a complex self built solution because it "feels" cheaper. If only I would have received a $ every time someone made a wrong decision based on a feeling I'd be a rich man today.
True, enthusiastic devs are definitely bad at amortizing their own time’s cost.