Elasticsearch node crashes can cause data loss

51–52 of 52 posts

Re: Elasticsearch node crashes can cause data loss

#51

Post by lobster_johnson » Sun, May 03, 2015, 12:55 AM UTC

fizx wrote: post #50

lobster_johnson wrote: post #49

Earlier quoted context omitted.

Of course you need to be able to do full syncs, and the sync is not a problem. But one needs to solve the two challenges I have described: 1. Determine how to do an incremental update, given that only the tail of the stream of updated documents is missing. Not as simple as just counting. 2. Determine when you must give up and fall back to a full sync; this is when not just the tail is missing, and finding the differe…

> "My second point was that this — streaming a "non-lossy" database as a change log into one or more "lossy" ones — is such a common operation that it should be a solved problem. It certainly requires something more than a queue." This is almost exactly the cross-DC replication problem, which is a subject of active research. A changelog on the source side is only sort of helpful. It's useful to advise which rows may…

Merkle trees (as used for anti-entropy in Cassandra, Riak etc.) are only practical when both sides of the replication can speak them, though.

I wonder, are Merkle trees viable for continuous streaming replication, not just repair?

Re: Elasticsearch node crashes can cause data loss

#52

Post by fizx » Sun, May 03, 2015, 2:07 AM UTC

lobster_johnson wrote: post #51

fizx wrote: post #50

Earlier quoted context omitted.

> "My second point was that this — streaming a "non-lossy" database as a change log into one or more "lossy" ones — is such a common operation that it should be a solved problem. It certainly requires something more than a queue." This is almost exactly the cross-DC replication problem, which is a subject of active research. A changelog on the source side is only sort of helpful. It's useful to advise which rows may…

Merkle trees (as used for anti-entropy in Cassandra, Riak etc.) are only practical when both sides of the replication can speak them, though. I wonder, are Merkle trees viable for continuous streaming replication, not just repair?

You'd implement the merkle trees yourself in the application layer. Alternately, you could use hash lists. It'd be somewhat similar to how you'd implement geohashing. Let's say you just take a SHA-X of each row represented as a hex varchar, then do something like "SELECT COUNT(*) GROUP BY SUBSTR(`sha_column`, 0, n)". If there's a count mismatch, then drill down into it by checking the first two chars, the first three chars, etc. Materialize some of these views as needed. It's ugly and tricky to tune.

Merkle trees aren't interesting in the no-repair case, as the changelog is more direct and has no downside.

Post reply on HN