Elasticsearch node crashes can cause data loss • Hacker News Board

Re: Elasticsearch node crashes can cause data loss

#11

Post by rdtsc » Sat, May 02, 2015, 4:32 AM UTC

eclark wrote: post #9

Crashes of a program will not affect the data being written to disk if said data has been written into the FS cache (not using std::ostream::write or other in user space buffering). Dirty pages will eventually be written to disk even if the process dies un-cleanly. Only something that keeps the kernel from flushing to disk can keep the page from being eventually written out. ( driver bug, kernel bug, hardware failure…

I think Kyle was just going by the documentation. And that is often what he tests -- how does the reality compare to the claims in the documentation and marketing.

So given these claims:

> Per-Operation Persistence. Elasticsearch puts your data safety first. Document changes are recorded in transaction logs on multiple nodes in the cluster to minimize the chance of any data loss.

One would hope they at least flushed the user space buffers.

Re: Elasticsearch node crashes can cause data loss

#12

Post by AnkhMorporkian » Sat, May 02, 2015, 4:44 AM UTC

bkeroack wrote: post #6

rdtsc wrote: post #4

Mandatory reading -- Last year's Call Me Maybe : Elasticsearch https://aphyr.com/posts/317-call-me-maybe-elasticsearch I've been hearing a lot of people talk about Elasticsearch lately. I get the same gut feeling I was getting about MongoDB back during the "Webscale" days.

In my experience, Elasticsearch is the single most common source of infrastructure downtime and service failure. It's basically my arch nemesis.

Really? Perhaps I was never running it at a large enough scale, but even pre-v1.0 I've basically never had any troubles with it (outside of operation concerns like occasionally confusing query syntax.) Then again, I never had more than 11 servers in the cluster so again I may just have never run into problems at scale.

Re: Elasticsearch node crashes can cause data loss

#13

Post by thejosh » Sat, May 02, 2015, 5:14 AM UTC

rdtsc wrote: post #4

Mandatory reading -- Last year's Call Me Maybe : Elasticsearch https://aphyr.com/posts/317-call-me-maybe-elasticsearch I've been hearing a lot of people talk about Elasticsearch lately. I get the same gut feeling I was getting about MongoDB back during the "Webscale" days.

I use ES only for search (indexes from a DB), so losing data isn't a massive drama, it's great for my usecase.

Re: Elasticsearch node crashes can cause data loss

#14

Post by rdtsc » Sat, May 02, 2015, 5:24 AM UTC

thejosh wrote: post #13

rdtsc wrote: post #4

Mandatory reading -- Last year's Call Me Maybe : Elasticsearch https://aphyr.com/posts/317-call-me-maybe-elasticsearch I've been hearing a lot of people talk about Elasticsearch lately. I get the same gut feeling I was getting about MongoDB back during the "Webscale" days.

I use ES only for search (indexes from a DB), so losing data isn't a massive drama, it's great for my usecase.

That sounds like the indended use. I should qualify my comment, I heard it advocated for a primary data storage.

Re: Elasticsearch node crashes can cause data loss

#15

Post by manigandham » Sat, May 02, 2015, 5:37 AM UTC

I'll just leave this here: one of my first answers on Quora about why ElasticSearch should not be a primary data store:

https://www.quora.com/Why-should-I-NOT-use-ElasticSearch-as-...

Re: Elasticsearch node crashes can cause data loss

#16

Post by elchief » Sat, May 02, 2015, 5:46 AM UTC

Who cares?

As long as we can cash in our options before the lawsuits come in, we win! Just like my last job in finance, actually.

Plus SQL's gross. You can't even webscale with it, and old people like it, so it must suck.

Re: Elasticsearch node crashes can cause data loss

#17

Post by po » Sat, May 02, 2015, 6:01 AM UTC

tedchs wrote: post #2

The advice I've heard from serious people using Elasticsearch for serious things indicate that you should definitely not use Elasticsearch as a primary data store (i.e. it should be treated as a cache).

It is often advocated as a datastore for logging data... which means (in that case) it's usually the primary datastore but perhaps not mission-critical.

Re: Elasticsearch node crashes can cause data loss

#18

Post by smegel » Sat, May 02, 2015, 6:27 AM UTC

Funnily enough I have seen a slew of technical bulletins from Cloudera warning of similar issues with HDFS.

Maybe not so funny if your multiply redundant cluster loses data because a single node dies...

Re: Elasticsearch node crashes can cause data loss

#19

Post by tomjen3 » Sat, May 02, 2015, 7:20 AM UTC

tedchs wrote: post #2

The advice I've heard from serious people using Elasticsearch for serious things indicate that you should definitely not use Elasticsearch as a primary data store (i.e. it should be treated as a cache).

It would probably be good enough as a store for A/B testing information - losing data here isn't critical but writing speed is.

Re: Elasticsearch node crashes can cause data loss

#20

Post by teraflop » Sat, May 02, 2015, 7:28 AM UTC

smegel wrote: post #18

Funnily enough I have seen a slew of technical bulletins from Cloudera warning of similar issues with HDFS. Maybe not so funny if your multiply redundant cluster loses data because a single node dies...

Wow, that sounds bad and I don't remember hearing about it. Do you have any pointers to bug reports or descriptions of the problem?

HDFS uses chain replication, so I would have expected that by the time the client got acknowledgement of a write, it would already be acknowledged by all replicas (3 by default). So even if there's a bug causing one of the nodes to go down without fsyncing, there shouldn't be any actual data loss.