Someone posted a comment on an open issue, I got the mail, but there's nothing on the web interface. What I don't understand is why they don't set it all to "read only" until the problem is sorted. Looks like any update goes to /dev/null, just let your users know! (unless they plan to replay those updates, somehow)
Status.github.com: “We're failing over a data storage system”
111–120 of 173 posts
Re: Status.github.com: “We're failing over a data storage system”
#112I don't know if they keep changing the text updates with a slightly different version to - prove it's a human that typed it - there is code the prevents repeating twice the same message either way it's entertaining... But it's Monday morning in Australia and we need to release! (yep we do this via pr/tagging etc.)
It is to ensure that the updates get mirrored onto twitter, where exact duplicates can't be posted.
Re: Status.github.com: “We're failing over a data storage system”
#113Earlier quoted context omitted.
I’m receiving emails (a lot of duplicates) from comments in PR, but they won’t show up in the browser. I guess people are trying to submit multiple times, the email is sent but comment isn’t posted.
Welcome to eventual consistency.
Re: Status.github.com: “We're failing over a data storage system”
#114> At 10:52 pm Sunday UTC, multiple services on GitHub.com were affected by a network partition and subsequent database failure resulting in inconsistent information being presented on our website. Out of an abundance of caution we have taken steps to ensure the integrity of your data, including pausing webhook events and other internal processing systems.
> We are aware of how important our services are to your development workflows and are actively working to establish an estimated timeframe for full recovery. We will share this information with you as soon as it is available. During this time, information displayed on GitHub.com is likely to appear out of date; however no data was lost. Once service is fully restored, everything should appear as expected. Further, this incident only impacted website metadata stored in our MySQL databases, such as issues and pull requests. Git repository data remains unaffected and has been available throughout the incident.
> We will continue to provide updates and an estimated time to resolution via our status page.
Re: Status.github.com: “We're failing over a data storage system”
#115Just a few weeks ago my organization was in the position of choosing a version control platform for our repos. I'm so glad we went ahead with self hosted gitlab. We installed it on a CentOS server at our premise, SSL'd via Let's encrypt, I've even set up a dedicated gitlab runner to use Gitlab CI for continuous delivery and so far the testing is progressing pretty smoothly. All this for $0. Update: I agree, gitlab an…
I feel like this is the first chapter of a story whose ending is probably happy, but whose plot twist is "and then the on-prem server crashed and it took a week to figure out how to fix it". Self-hosting vs 3rd party hosting has a ton of tradeoffs, but systems can fall over no matter who's hosting them.
With hard drive failure rates of 1.6% per year, and all other bits of computer hardware being very reliable, I would guess that an unmaintained self hosted solution will have a longer 'time to failure' than a service provider.
The main benefit of a hosted service IMO is less setup time and a support team to help you when they inevitably 'retire' the 'old' api.
Re: Status.github.com: “We're failing over a data storage system”
#116> Multiple services on GitHub.com were affected by a network partition and subsequent database failure resulting in inconsistent information being presented on our website. Out of an abundance of caution we have taken steps to ensure the integrity of your data, including pausing webhook events and other internal processing systems.
[...]
> Information displayed on GitHub.com is likely to appear out of date; however no data was lost. Once service is fully restored, everything should appear as expected. Further, this incident only impacted website metadata stored in our MySQL databases, such as issues and pull requests. Git repository data remains unaffected and has been available throughout the incident.
Re: Status.github.com: “We're failing over a data storage system”
#117Someone posted a comment on an open issue, I got the mail, but there's nothing on the web interface. What I don't understand is why they don't set it all to "read only" until the problem is sorted. Looks like any update goes to /dev/null, just let your users know! (unless they plan to replay those updates, somehow)
If replaying is even on the table (which sounds very dangerous to me), that requires a huge coordination and I am pretty sure they are not able to tell right now if that is going to work once the actual issue is fixed.
Better to stay quiet while fixing the issue and only say things that you actually know 100% to be correct.
Re: Status.github.com: “We're failing over a data storage system”
#118Re: Status.github.com: “We're failing over a data storage system”
#119I don't know if they keep changing the text updates with a slightly different version to - prove it's a human that typed it - there is code the prevents repeating twice the same message either way it's entertaining... But it's Monday morning in Australia and we need to release! (yep we do this via pr/tagging etc.)
Re: Status.github.com: “We're failing over a data storage system”
#120Earlier quoted context omitted.
yes, but does it scale tho ? is it DR ? I mean sure fine it's easy and pretty but there is some operation challenge to figure out,
If you host just your own Open Source projects, does it need to scale?