Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Elasticsearch node crashes can cause data loss

github.com

Post reply on HN
Print view View original

41–50 of 52 posts

Re: Elasticsearch node crashes can cause data loss

#41

Post by capkutay »

m-i-l wrote: post #39
digitalzombie wrote: post #23

Earlier quoted context omitted.

Elasticsearch is just a text search engine base on lucene. You either use ES, Solr, or Lucene library if you want fuzzy search and such. You really want to use it in tandem with a storage db PostgreSQL, Cassandra, MongDB. Where ES or any lucene based indexer/db would be use for text searching. I personally like PostgreSQL and Cassandra, would use it in tadem with ES. Solr, last I check was a bit complicated to cluste…

> Solr, last I check was a bit complicated to cluster SolrCloud, with Zookeeper, is relatively new and not too difficult to set up.

Does it still have the issue where you have to take the cluster down to create a new index or modify existing ones?

Re: Elasticsearch node crashes can cause data loss

#42

Post by flippyhead »

bkeroack wrote: post #6
rdtsc wrote: post #4

Mandatory reading -- Last year's Call Me Maybe : Elasticsearch https://aphyr.com/posts/317-call-me-maybe-elasticsearch I've been hearing a lot of people talk about Elasticsearch lately. I get the same gut feeling I was getting about MongoDB back during the "Webscale" days.

In my experience, Elasticsearch is the single most common source of infrastructure downtime and service failure. It's basically my arch nemesis.

While I don't necessarily disagree, I do find that this depends entirely on how ES is used. All too often people dive headfirst into using elastic search in ways it really should not be used.

Re: Elasticsearch node crashes can cause data loss

#44

Post by sitkack »

eternalban wrote: post #38
sitkack wrote: post #34

Earlier quoted context omitted.

Fsync should always be on by default. Require the user to turn it off. I'd even argue that `fsync` itself is broken and that semantics should be inverted.

What about pervasive virtualization? The issue here is not really fsync. A fault-tolerant in-memory cluster should not lose data.

Writes should be durable by whatever means the platform deems durable. Durable should be the default. It should take work to have non-durable io.

Re: Elasticsearch node crashes can cause data loss

#45

Post by bkeroack »

willejs wrote: post #32
bkeroack wrote: post #6

Earlier quoted context omitted.

In my experience, Elasticsearch is the single most common source of infrastructure downtime and service failure. It's basically my arch nemesis.

I am interested to hear a bit more about this, as I find it hard to believe. I have only ran it at pretty small scale - x8 servers, around 300 million documents indexed a day, peak index rate 30k docs/sec. I found that you have to monitor it correctly, tune the JVM slightly (Mostly GC), give it fast disks, lots of ram, and the correct architecture (search, index & data nodes) to get the most out of it. Once I did tha…

The full explanation deserves a blog post, but in a nutshell it revolves around the issue that ES contains a huge amount of complexity around a feature that is actually fairly useless (the "elastic" part) or at least difficult to use correctly. I've found that you need to be a deep expert in ES to architect and run it properly (or have access to such expertise) and even then it requires regular care and feeding to maintain uptime. In a short-deadline startup world you probably won't have time for any of that--once it's working it will lull you into a false sense of security and then completely blow up a few weeks/months later.

Re: Elasticsearch node crashes can cause data loss

#46

Post by fizx »

lobster_johnson wrote: post #8
fizx wrote: post #7

Earlier quoted context omitted.

A logical "SELECT COUNT(*) WHERE updated_at < now()" is probably reasonably fast on your primary store and ElasticSearch.

Given that ElasticSearch is "eventually consistent", how do you know when it has caught up? How do you know what data is missing once the count is wrong? It's solveable, of course, but it's a problem that pops up with any synchronization system, and I'm surprised nobody (apparently) has written one, because it requires a fairly good state machine that can also compute diffs. Once a store grows to a certain state, you…

> Given that ElasticSearch is "eventually consistent", how do you know when it has caught up?

That's nice, in practice, though, ElasticSearch, doesn't behave like an eventually consistent system--it behaves like a flawed fully consistent system. It doesn't self-repair enough to be eventually consistent. If you get out of sync by more than a few seconds, you're going to have to repair the system manually in some fashion. It never "catches up."

Additionally, also in practice, most of the data loss (and real eventual consistency behavior) you'll see in an ElasticSearch+primary-data-store system isn't coming from within ES--it's coming from queues people typically use in the sync process. So there's a degree where you're going to need to handle this on an application-specific basis.

> How do you know what data is missing once the count is wrong?

In practice, people just ignore it or do a full re-index. Theoretically, you should be building merkle trees.

> Once a store grows to a certain state, you do not want to trigger full syncs, ever.

This is not really true. You need to maintain enough capacity for full syncs, because someone will need to do schema changes and/or change linguistic features in the search index.

Re: Elasticsearch node crashes can cause data loss

#47

Post by thecage411 »

capkutay wrote: post #41
m-i-l wrote: post #39

Earlier quoted context omitted.

> Solr, last I check was a bit complicated to cluster SolrCloud, with Zookeeper, is relatively new and not too difficult to set up.

Does it still have the issue where you have to take the cluster down to create a new index or modify existing ones?

No, search for MergeIndexes or --go-live.

Re: Elasticsearch node crashes can cause data loss

#48

Post by lobster_johnson »

bkeroack wrote: post #6
rdtsc wrote: post #4

Mandatory reading -- Last year's Call Me Maybe : Elasticsearch https://aphyr.com/posts/317-call-me-maybe-elasticsearch I've been hearing a lot of people talk about Elasticsearch lately. I get the same gut feeling I was getting about MongoDB back during the "Webscale" days.

In my experience, Elasticsearch is the single most common source of infrastructure downtime and service failure. It's basically my arch nemesis.

It can't be worse than RabbitMQ... can it?

Re: Elasticsearch node crashes can cause data loss

#49

Post by lobster_johnson »

fizx wrote: post #46
lobster_johnson wrote: post #8

Earlier quoted context omitted.

Given that ElasticSearch is "eventually consistent", how do you know when it has caught up? How do you know what data is missing once the count is wrong? It's solveable, of course, but it's a problem that pops up with any synchronization system, and I'm surprised nobody (apparently) has written one, because it requires a fairly good state machine that can also compute diffs. Once a store grows to a certain state, you…

> Given that ElasticSearch is "eventually consistent", how do you know when it has caught up? That's nice, in practice, though, ElasticSearch, doesn't behave like an eventually consistent system--it behaves like a flawed fully consistent system. It doesn't self-repair enough to be eventually consistent. If you get out of sync by more than a few seconds, you're going to have to repair the system manually in some fashi…

Of course you need to be able to do full syncs, and the sync is not a problem. But one needs to solve the two challenges I have described:

1. Determine how to do an incremental update, given that only the tail of the stream of updated documents is missing. Not as simple as just counting.

2. Determine when you must give up and fall back to a full sync; this is when not just the tail is missing, and finding the difference is computationally non-trivial. You'll only want to do this once you're sure that you need to.

My point remains that ElasticSearch's consistency model means it's hard to even do #1, which is the day-to-day streaming updates.

My second point was that this — streaming a "non-lossy" database as a change log into one or more "lossy" ones — is such a common operation that it should be a solved problem. It certainly requires something more than a queue.

(In my experience, queues are terrible at this. One problem is that it's hard to express different priorities this way. If you have a batch job that touches 1 million database rows, you don't want these to fill your "real-time" queue with pending indexing operations. Using multiple queues leaves you open to odd inconsistencies when updates are applied out of order. And so on. Polling triggered by notifications tends to be better.)

Re: Elasticsearch node crashes can cause data loss

#50

Post by fizx »

lobster_johnson wrote: post #49
fizx wrote: post #46

Earlier quoted context omitted.

> Given that ElasticSearch is "eventually consistent", how do you know when it has caught up? That's nice, in practice, though, ElasticSearch, doesn't behave like an eventually consistent system--it behaves like a flawed fully consistent system. It doesn't self-repair enough to be eventually consistent. If you get out of sync by more than a few seconds, you're going to have to repair the system manually in some fashi…

Of course you need to be able to do full syncs, and the sync is not a problem. But one needs to solve the two challenges I have described: 1. Determine how to do an incremental update, given that only the tail of the stream of updated documents is missing. Not as simple as just counting. 2. Determine when you must give up and fall back to a full sync; this is when not just the tail is missing, and finding the differe…

> "My second point was that this — streaming a "non-lossy" database as a change log into one or more "lossy" ones — is such a common operation that it should be a solved problem. It certainly requires something more than a queue."

This is almost exactly the cross-DC replication problem, which is a subject of active research.

A changelog on the source side is only sort of helpful. It's useful to advise which rows may have changed, but given that you don't trust the target database, you also need to do repair.

Correct repair is impossible without full syncs (or at least partial sync since known-perfectly-synced snapshot), unless your data model is idempotent and commutative. On-the-fly repair requires out-of-order re-application of previous operations.

The easiest way to reason about commutivity is to just make everything in your database immutable. So this is a solved problem, but it requires compromises in the data model that people are mostly unwilling to live with.

You can do pretty well if your target database supports idempotent operations.

If you're trying to do pretty well, then you can do a Merkle Tree comparison of both sides at some time (T0 = now() - epsilon) to efficiently search into which records have been lost or misplaced. Then you re-sync them. Here, for efficency, your merkle tree implementation will ideally to span field(s) that are related to the updated_at time, so that only a small subset of the tree is changing all the time. This is a tricky thing to tune.

You'll still be "open to odd inconsistencies when updates are applied out of order" if you haven't made your data model immutable, but I think this is mostly inline with your hopes.

Post reply on HN