Elasticsearch node crashes can cause data loss
1–10 of 52 posts
Re: Elasticsearch node crashes can cause data loss
#2The advice I've heard from serious people using Elasticsearch for serious things indicate that you should definitely not use Elasticsearch as a primary data store (i.e. it should be treated as a cache).
Re: Elasticsearch node crashes can cause data loss
#3Do I understand it correctly from skimming aphyr's article https://aphyr.com/posts/317-call-me-maybe-elasticsearch that TL;DR: Elasticsearch does not use a WAL with a real consensus algorithm such as Paxos or Raft, and therefore isn't reliable?
Re: Elasticsearch node crashes can cause data loss
#4Mandatory reading -- Last year's Call Me Maybe : Elasticsearch
https://aphyr.com/posts/317-call-me-maybe-elasticsearch
I've been hearing a lot of people talk about Elasticsearch lately. I get the same gut feeling I was getting about MongoDB back during the "Webscale" days.
Re: Elasticsearch node crashes can cause data loss
#5The advice I've heard from serious people using Elasticsearch for serious things indicate that you should definitely not use Elasticsearch as a primary data store (i.e. it should be treated as a cache).
This is true. On the other hand, even a secondary data store that's considered "lossy" poses a challenge — how do you know if its integrity has been compromised?
In other words, if you're firehosing your primary data store into ElasticSearch, you'll want to know whether it's got all the data you pushed to it at any given time.
I suppose you could use some kind of heuristic to detect this, like posting a "checksum" document occasionally that contains the indexing state and thus acts as a canary that lets you detect loss. On the other hand, this document would be sharded, so you'd want one such document per shard. Is this a solved problem?
Re: Elasticsearch node crashes can cause data loss
#6Mandatory reading -- Last year's Call Me Maybe : Elasticsearch https://aphyr.com/posts/317-call-me-maybe-elasticsearch I've been hearing a lot of people talk about Elasticsearch lately. I get the same gut feeling I was getting about MongoDB back during the "Webscale" days.
In my experience, Elasticsearch is the single most common source of infrastructure downtime and service failure. It's basically my arch nemesis.
Re: Elasticsearch node crashes can cause data loss
#7The advice I've heard from serious people using Elasticsearch for serious things indicate that you should definitely not use Elasticsearch as a primary data store (i.e. it should be treated as a cache).
This is true. On the other hand, even a secondary data store that's considered "lossy" poses a challenge — how do you know if its integrity has been compromised? In other words, if you're firehosing your primary data store into ElasticSearch, you'll want to know whether it's got all the data you pushed to it at any given time. I suppose you could use some kind of heuristic to detect this, like posting a "checksum" do…
A logical "SELECT COUNT(*) WHERE updated_at < now()" is probably reasonably fast on your primary store and ElasticSearch.
Re: Elasticsearch node crashes can cause data loss
#8Earlier quoted context omitted.
This is true. On the other hand, even a secondary data store that's considered "lossy" poses a challenge — how do you know if its integrity has been compromised? In other words, if you're firehosing your primary data store into ElasticSearch, you'll want to know whether it's got all the data you pushed to it at any given time. I suppose you could use some kind of heuristic to detect this, like posting a "checksum" do…
A logical "SELECT COUNT(*) WHERE updated_at < now()" is probably reasonably fast on your primary store and ElasticSearch.
Given that ElasticSearch is "eventually consistent", how do you know when it has caught up? How do you know what data is missing once the count is wrong?
It's solveable, of course, but it's a problem that pops up with any synchronization system, and I'm surprised nobody (apparently) has written one, because it requires a fairly good state machine that can also compute diffs. Once a store grows to a certain state, you do not want to trigger full syncs, ever.
The best, most trivial (in terms of complexity and resilience) solution I have found is to sync data in batches, give each batch an ID, and record each batch both in the target (eg., ElasticSearch) and in a log that belongs to the synchronization process. The heuristic is then to compute a difference beetween the two logs to see how far you need to catch up.
This will only work in a sequential fashion; if ElasticSearch loses random documents, it won't be picked up by such a system. You could fix this by letting each log entry store the list of logical updates, checksummed; and then do regular (eg., nightly) "inventory checks".
Re: Elasticsearch node crashes can cause data loss
#9Crashes of a program will not affect the data being written to disk if said data has been written into the FS cache (not using std::ostream::write or other in user space buffering). Dirty pages will eventually be written to disk even if the process dies un-cleanly. Only something that keeps the kernel from flushing to disk can keep the page from being eventually written out. ( driver bug, kernel bug, hardware failure, power failure ).
From reading the code in Jepsen it looks like kill -9 is all that's being used to start failures. So there's a real bug here: https://github.com/aphyr/jepsen/blob/master/elasticsearch/sr...
Re: Elasticsearch node crashes can cause data loss
#10[deleted]