Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Elasticsearch node crashes can cause data loss

github.com

Post reply on HN
Print view View original

21–30 of 52 posts

Re: Elasticsearch node crashes can cause data loss

#21

Post by alrs »

po wrote: post #17
tedchs wrote: post #2

The advice I've heard from serious people using Elasticsearch for serious things indicate that you should definitely not use Elasticsearch as a primary data store (i.e. it should be treated as a cache).

It is often advocated as a datastore for logging data... which means (in that case) it's usually the primary datastore but perhaps not mission-critical.

It's a great index for log data.

Spew your log data into a standard syslog server, while also pumping it into Logstash.

Using Elasticsearch as your canonical log storage would be ridiculous.

Re: Elasticsearch node crashes can cause data loss

#22

Post by digitalzombie »

Title should have ElasticSearch in there...

I was thinking of NodeJS.

But the comment is correct, ES is not a db but a indexer and search engine.

edit:

Oh god, don't use it for storage. It index stuff.

You got a document, it'll store it in root words form so you can fuzzy search. It'll also do other NLP stuff to your document such as removing stop words. Once it hit an index you can store index value that point to your primary storage (cassandra, postgresql).

At least that's how I used it. If there is any better alternative I'd like to know about it.

edit:

I highly recommend: http://www.manning.com/ingersoll/

Taming text by Ingersoll and it won Dr. Dobb award too as a good book.

Re: Elasticsearch node crashes can cause data loss

#23

Post by digitalzombie »

rdtsc wrote: post #4

Mandatory reading -- Last year's Call Me Maybe : Elasticsearch https://aphyr.com/posts/317-call-me-maybe-elasticsearch I've been hearing a lot of people talk about Elasticsearch lately. I get the same gut feeling I was getting about MongoDB back during the "Webscale" days.

Elasticsearch is just a text search engine base on lucene. You either use ES, Solr, or Lucene library if you want fuzzy search and such.

You really want to use it in tandem with a storage db PostgreSQL, Cassandra, MongDB. Where ES or any lucene based indexer/db would be use for text searching.

I personally like PostgreSQL and Cassandra, would use it in tadem with ES. Solr, last I check was a bit complicated to cluster.

Re: Elasticsearch node crashes can cause data loss

#24

Post by digitalzombie »

rdtsc wrote: post #14
thejosh wrote: post #13

Earlier quoted context omitted.

I use ES only for search (indexes from a DB), so losing data isn't a massive drama, it's great for my usecase.

That sounds like the indended use. I should qualify my comment, I heard it advocated for a primary data storage.

Yeah, I had an argument with that over at reddit. Where someone advocated ES as an alternative to Cassandra. >___<. I did hopefully, convinced the user otherwise.

Re: Elasticsearch node crashes can cause data loss

#25

Post by dang »

digitalzombie wrote: post #22

Title should have ElasticSearch in there... I was thinking of NodeJS. But the comment is correct, ES is not a db but a indexer and search engine. edit: Oh god, don't use it for storage. It index stuff. You got a document, it'll store it in root words form so you can fuzzy search. It'll also do other NLP stuff to your document such as removing stop words. Once it hit an index you can store index value that point to yo…

That was our mistake. We put it back. Thanks.

Re: Elasticsearch node crashes can cause data loss

#26

Post by EdwardDiego »

teraflop wrote: post #20
smegel wrote: post #18

Funnily enough I have seen a slew of technical bulletins from Cloudera warning of similar issues with HDFS. Maybe not so funny if your multiply redundant cluster loses data because a single node dies...

Wow, that sounds bad and I don't remember hearing about it. Do you have any pointers to bug reports or descriptions of the problem? HDFS uses chain replication, so I would have expected that by the time the client got acknowledgement of a write, it would already be acknowledged by all replicas (3 by default). So even if there's a bug causing one of the nodes to go down without fsyncing, there shouldn't be any actual…

Yeah, I'm very interested in this also.

Re: Elasticsearch node crashes can cause data loss

#27

Post by growse »

teraflop wrote: post #20
smegel wrote: post #18

Funnily enough I have seen a slew of technical bulletins from Cloudera warning of similar issues with HDFS. Maybe not so funny if your multiply redundant cluster loses data because a single node dies...

Wow, that sounds bad and I don't remember hearing about it. Do you have any pointers to bug reports or descriptions of the problem? HDFS uses chain replication, so I would have expected that by the time the client got acknowledgement of a write, it would already be acknowledged by all replicas (3 by default). So even if there's a bug causing one of the nodes to go down without fsyncing, there shouldn't be any actual…

I think the client assumes the data is written after $dfs.namenode.replication.min blocks have been written, which I think is 1 by default.

What it actually means inside HDFS when it claims 'written', I'm not sure - I'd assume flushed to the dirty page buffer at a minimum and would hope fsync.

Re: Elasticsearch node crashes can cause data loss

#28

Post by riceo100 »

bkeroack wrote: post #6
rdtsc wrote: post #4

Mandatory reading -- Last year's Call Me Maybe : Elasticsearch https://aphyr.com/posts/317-call-me-maybe-elasticsearch I've been hearing a lot of people talk about Elasticsearch lately. I get the same gut feeling I was getting about MongoDB back during the "Webscale" days.

In my experience, Elasticsearch is the single most common source of infrastructure downtime and service failure. It's basically my arch nemesis.

Same here. A single node failure has lead to the whole cluster crashing down around me on more than one occasion.

Re: Elasticsearch node crashes can cause data loss

#29

Post by smegel »

teraflop wrote: post #20
smegel wrote: post #18

Funnily enough I have seen a slew of technical bulletins from Cloudera warning of similar issues with HDFS. Maybe not so funny if your multiply redundant cluster loses data because a single node dies...

Wow, that sounds bad and I don't remember hearing about it. Do you have any pointers to bug reports or descriptions of the problem? HDFS uses chain replication, so I would have expected that by the time the client got acknowledgement of a write, it would already be acknowledged by all replicas (3 by default). So even if there's a bug causing one of the nodes to go down without fsyncing, there shouldn't be any actual…

>>> OK its not simply that a node dies, but that disks on a node are replaced (which might sort of be related to a node dying).

TSB 2015-51: Replacing DataNode Disks or Manually changing the Storage IDs of Volumes in a Cluster may result in Data Loss Printable View Rate This Knowledge Article (Average Rating: 3.3) Show Properties « Go Back Information

Purpose Updated: 4/22/2015

In CDH 4, DataNodes are identified in HDFS with a single unique identifier. Beginning with CDH 5, every individual disk in a DataNode is assigned a unique identifier as well.

A bug discovered in HDFS, HDFS-7960, can result in the NameNode improperly accounting for DataNode storages for which Storage IDs have changed. A Storage ID changes whenever a disk on a DataNode is replaced, or if the Storage ID is manually manipulated. Either of these scenarios causes the NameNode to double-count block replicas, incorrectly determine that a block is over-replicated, and remove those replicas permanently from those DataNodes.

A related bug, HDFS-7575, results in a failure to create unique IDs for each disk within the DataNodes during upgrade from CDH 4 to CDH 5. Instead, all disks within a single DataNode are assigned the same ID. This bug by itself negatively impacts proper function of the HDFS balancer. Cloudera Release Notes originally stated that manually changing the Storage IDs of the DataNodes was a valid workaround for HDFS-7575. However, doing so can result in irrecoverable data loss due to HDFS-7960, and the release notes have been corrected.

Users affected:

Any cluster where Storage IDs change can be affected by HDFS-7960. Storage IDs change whenever a disk is replaced, or when Storage IDs are manually manipulated. Only clusters upgraded from CDH 4 or earlier releases are affected by HDFS-7575.

Symptoms If data loss has occurred, the NameNode reports “missing blocks” on the NameNode Web UI. You can determine to which files the missing blocks belong by using FSCK. You can also search for NameNode log lines like the following, which indicate that a Storage ID has changed and data loss may have occurred: 2015-03-21 06:48:02,556 WARN BlockStateChange: BLOCK* addStoredBlock: Redundant addStoredBlock request received for blk_8271694345820118657_530878393 on 10.11.12.13:1004 size 6098 Impact:

The replacement of DataNode disks, or manual manipulation of DataNode Storage IDs, can result in irrecoverable data loss. Additionally, due to HDFS-7575, the HDFS Balancer will not function properly.

Applies To HDFS All CDH 5 releases prior 3/31/15, including: 5.0, 5.0.1, 5.0.2, 5.0.3, 5.0.4, 5.0.5 5.1, 5.1.2, 5.1.3, 5.1.4 5.2, 5.2.1, 5.2.3, 5.2.4 5.3, 5.3.1, 5.3.2 Cause Instructions Immediate action required:

Do not manually manipulate Storage IDs on DataNode disks. Additionally, do not replace failed DataNode disks when running any of the affected CDH versions.

Upgrade to CDH 5.4.0, 5.3.3, 5.2.5, 5.1.5, or 5.0.6. See Also/Related Articles Apache.org Bug HDFS-7575

HDFS-7960 Attachment

Re: Elasticsearch node crashes can cause data loss

#30

Post by threeseed »

digitalzombie wrote: post #23
rdtsc wrote: post #4

Mandatory reading -- Last year's Call Me Maybe : Elasticsearch https://aphyr.com/posts/317-call-me-maybe-elasticsearch I've been hearing a lot of people talk about Elasticsearch lately. I get the same gut feeling I was getting about MongoDB back during the "Webscale" days.

Elasticsearch is just a text search engine base on lucene. You either use ES, Solr, or Lucene library if you want fuzzy search and such. You really want to use it in tandem with a storage db PostgreSQL, Cassandra, MongDB. Where ES or any lucene based indexer/db would be use for text searching. I personally like PostgreSQL and Cassandra, would use it in tadem with ES. Solr, last I check was a bit complicated to cluste…

Agreed. Cassandra is especially nice if you have the DataStax Enterprise version which allows for seamless integration between the two.
Post reply on HN