Live data from Hacker News

Status.github.com: “We're failing over a data storage system”

news.ycombinator.com

111–120 of 173 posts

Re: Status.github.com: “We're failing over a data storage system”

#111

Someone posted a comment on an open issue, I got the mail, but there's nothing on the web interface. What I don't understand is why they don't set it all to "read only" until the problem is sorted. Looks like any update goes to /dev/null, just let your users know! (unless they plan to replay those updates, somehow)

[deleted]

Re: Status.github.com: “We're failing over a data storage system”

#112
post #32
post #20

I don't know if they keep changing the text updates with a slightly different version to - prove it's a human that typed it - there is code the prevents repeating twice the same message either way it's entertaining... But it's Monday morning in Australia and we need to release! (yep we do this via pr/tagging etc.)

It is to ensure that the updates get mirrored onto twitter, where exact duplicates can't be posted.

If you include a unique hashtag, would the message be considered unique? In that case, you could include the epoch timestamp as a hashtag :-)

Re: Status.github.com: “We're failing over a data storage system”

#113

Earlier quoted context omitted.

I’m receiving emails (a lot of duplicates) from comments in PR, but they won’t show up in the browser. I guess people are trying to submit multiple times, the email is sent but comment isn’t posted.

Welcome to eventual consistency.

Mañana Consistency(tm)

Re: Status.github.com: “We're failing over a data storage system”

#114
An update has been posted there: https://blog.github.com/2018-10-21-october21-incident-report...

> At 10:52 pm Sunday UTC, multiple services on GitHub.com were affected by a network partition and subsequent database failure resulting in inconsistent information being presented on our website. Out of an abundance of caution we have taken steps to ensure the integrity of your data, including pausing webhook events and other internal processing systems.

> We are aware of how important our services are to your development workflows and are actively working to establish an estimated timeframe for full recovery. We will share this information with you as soon as it is available. During this time, information displayed on GitHub.com is likely to appear out of date; however no data was lost. Once service is fully restored, everything should appear as expected. Further, this incident only impacted website metadata stored in our MySQL databases, such as issues and pull requests. Git repository data remains unaffected and has been available throughout the incident.

> We will continue to provide updates and an estimated time to resolution via our status page.

Re: Status.github.com: “We're failing over a data storage system”

#115
post #96
post #91

Just a few weeks ago my organization was in the position of choosing a version control platform for our repos. I'm so glad we went ahead with self hosted gitlab. We installed it on a CentOS server at our premise, SSL'd via Let's encrypt, I've even set up a dedicated gitlab runner to use Gitlab CI for continuous delivery and so far the testing is progressing pretty smoothly. All this for $0. Update: I agree, gitlab an…

I feel like this is the first chapter of a story whose ending is probably happy, but whose plot twist is "and then the on-prem server crashed and it took a week to figure out how to fix it". Self-hosting vs 3rd party hosting has a ton of tradeoffs, but systems can fall over no matter who's hosting them.

"and then the service we were using got 'sunsetted' and we spent weeks migrating to a new incompatible provider".

With hard drive failure rates of 1.6% per year, and all other bits of computer hardware being very reliable, I would guess that an unmaintained self hosted solution will have a longer 'time to failure' than a service provider.

The main benefit of a hosted service IMO is less setup time and a support team to help you when they inevitably 'retire' the 'old' api.

Re: Status.github.com: “We're failing over a data storage system”

#116
They've just posted an "Incident Report": https://blog.github.com/2018-10-21-october21-incident-report...

> Multiple services on GitHub.com were affected by a network partition and subsequent database failure resulting in inconsistent information being presented on our website. Out of an abundance of caution we have taken steps to ensure the integrity of your data, including pausing webhook events and other internal processing systems.

[...]

> Information displayed on GitHub.com is likely to appear out of date; however no data was lost. Once service is fully restored, everything should appear as expected. Further, this incident only impacted website metadata stored in our MySQL databases, such as issues and pull requests. Git repository data remains unaffected and has been available throughout the incident.

Re: Status.github.com: “We're failing over a data storage system”

#117

Someone posted a comment on an open issue, I got the mail, but there's nothing on the web interface. What I don't understand is why they don't set it all to "read only" until the problem is sorted. Looks like any update goes to /dev/null, just let your users know! (unless they plan to replay those updates, somehow)

I don't think it's easy to predict what is going to happen to all updates now. They run a big distributed infrastructure. I doubt they can even predict which updates will be commited and which ones will not.

If replaying is even on the table (which sounds very dangerous to me), that requires a huge coordination and I am pretty sure they are not able to tell right now if that is going to work once the actual issue is fixed.

Better to stay quiet while fixing the issue and only say things that you actually know 100% to be correct.

Re: Status.github.com: “We're failing over a data storage system”

#119
post #20

I don't know if they keep changing the text updates with a slightly different version to - prove it's a human that typed it - there is code the prevents repeating twice the same message either way it's entertaining... But it's Monday morning in Australia and we need to release! (yep we do this via pr/tagging etc.)

https://github-status-generator.com/

Re: Status.github.com: “We're failing over a data storage system”

#120

Earlier quoted context omitted.

yes, but does it scale tho ? is it DR ? I mean sure fine it's easy and pretty but there is some operation challenge to figure out,

If you host just your own Open Source projects, does it need to scale?

Likely not, you can probably even host your friends too without much server load. Scaling is a concern when you have many users, just a handful of users won't ttend to create much load.
Post reply on HN