31–40 of 52 posts
The advice I've heard from serious people using Elasticsearch for serious things indicate that you should definitely not use Elasticsearch as a primary data store (i.e. it should be treated as a cache).
It is often advocated as a datastore for logging data... which means (in that case) it's usually the primary datastore but perhaps not mission-critical.
Mandatory reading -- Last year's Call Me Maybe : Elasticsearch https://aphyr.com/posts/317-call-me-maybe-elasticsearch I've been hearing a lot of people talk about Elasticsearch lately. I get the same gut feeling I was getting about MongoDB back during the "Webscale" days.
In my experience, Elasticsearch is the single most common source of infrastructure downtime and service failure. It's basically my arch nemesis.
> by not fsync'ing each operation (though one can configure ES to do so).
It may not be default, but we've seen, again and again, how people are influenced by what they read about a database (e.g. MongoDB).
The lesson by now should be: always know your DB.
Suggested reading -- the link. > by not fsync'ing each operation (though one can configure ES to do so). It may not be default, but we've seen, again and again, how people are influenced by what they read about a database (e.g. MongoDB). The lesson by now should be: always know your DB .
Earlier quoted context omitted.
I use ES only for search (indexes from a DB), so losing data isn't a massive drama, it's great for my usecase.
That sounds like the indended use. I should qualify my comment, I heard it advocated for a primary data storage.
Earlier quoted context omitted.
Wow, that sounds bad and I don't remember hearing about it. Do you have any pointers to bug reports or descriptions of the problem? HDFS uses chain replication, so I would have expected that by the time the client got acknowledgement of a write, it would already be acknowledged by all replicas (3 by default). So even if there's a bug causing one of the nodes to go down without fsyncing, there shouldn't be any actual…
>>> OK its not simply that a node dies, but that disks on a node are replaced (which might sort of be related to a node dying). TSB 2015-51: Replacing DataNode Disks or Manually changing the Storage IDs of Volumes in a Cluster may result in Data Loss Printable View Rate This Knowledge Article (Average Rating: 3.3) Show Properties « Go Back Information Purpose Updated: 4/22/2015 In CDH 4, DataNodes are identified in H…
That is not comparable to what seems to be the case here with ES & MongoDB, where they deliberately (by design) accept the risk of data-loss to boost performance. Now most systems allow you to do make that trade-off, but an honest system chooses the safe-but-slow configuration by default, and has you knowingly opt-in to the risks of the faster configuration.
I hope you consider editing your initial post - if you conflate bugs with deliberately unsafe design, we just end up with a race to the bottom of increasingly unsafe but fast behaviour.
Mandatory reading -- Last year's Call Me Maybe : Elasticsearch https://aphyr.com/posts/317-call-me-maybe-elasticsearch I've been hearing a lot of people talk about Elasticsearch lately. I get the same gut feeling I was getting about MongoDB back during the "Webscale" days.
Elasticsearch is just a text search engine base on lucene. You either use ES, Solr, or Lucene library if you want fuzzy search and such. You really want to use it in tandem with a storage db PostgreSQL, Cassandra, MongDB. Where ES or any lucene based indexer/db would be use for text searching. I personally like PostgreSQL and Cassandra, would use it in tadem with ES. Solr, last I check was a bit complicated to cluste…
Suggested reading -- the link. > by not fsync'ing each operation (though one can configure ES to do so). It may not be default, but we've seen, again and again, how people are influenced by what they read about a database (e.g. MongoDB). The lesson by now should be: always know your DB .
Fsync should always be on by default. Require the user to turn it off. I'd even argue that `fsync` itself is broken and that semantics should be inverted.
Mandatory reading -- Last year's Call Me Maybe : Elasticsearch https://aphyr.com/posts/317-call-me-maybe-elasticsearch I've been hearing a lot of people talk about Elasticsearch lately. I get the same gut feeling I was getting about MongoDB back during the "Webscale" days.
Elasticsearch is just a text search engine base on lucene. You either use ES, Solr, or Lucene library if you want fuzzy search and such. You really want to use it in tandem with a storage db PostgreSQL, Cassandra, MongDB. Where ES or any lucene based indexer/db would be use for text searching. I personally like PostgreSQL and Cassandra, would use it in tadem with ES. Solr, last I check was a bit complicated to cluste…
SolrCloud, with Zookeeper, is relatively new and not too difficult to set up.
Suggested reading -- the link. > by not fsync'ing each operation (though one can configure ES to do so). It may not be default, but we've seen, again and again, how people are influenced by what they read about a database (e.g. MongoDB). The lesson by now should be: always know your DB .
True, but Elasticsearch is not intended to be a permanent datastore.