Using Elasticsearch as the Primary Data Store in an ETL Pipeline • Hacker News Board

Re: Using Elasticsearch as the Primary Data Store in an ETL Pipeline

#41

Post by JohnBooty » Mon, Dec 03, 2018, 7:45 AM UTC

haggy wrote: post #16

kn7 wrote: post #14

Earlier quoted context omitted.

That still doesn't guarantee that a consecutive read is gonna get the last state.

Direct from ES documentation on the `refresh` flag: "Refresh the relevant primary and replica shards (not the whole index) immediately after the operation occurs, so that the updated document appears in search results immediately." Do you have other information about the refresh flag because their documentation clearly states that forcing a refresh is applied to primary and replica shards meaning that it will be avai…

My experience was that this was not reliable. It was nearly always true, but not always, and tests would sporadically fail some small percentage of the time.

However, this was back in ~2015 and Elasticsearch 1.3 or something like this, which is of course a now-ancient version. Perhaps things are different now.

edit: Perhaps we were using the refresh command and not the refresh flag. It was a few years ago and I don't have access to the code any more, and my memory may be failing here. If the refresh flag works as advertised (enforces an index update and guarantees a consistent view of the data for the next query, which the command did not seem to) then that of course solves my initial problem W.R.T. writing tests.

Re: Using Elasticsearch as the Primary Data Store in an ETL Pipeline

#42

Post by buckhx » Mon, Dec 03, 2018, 9:25 PM UTC

simonw wrote: post #32

buckhx wrote: post #18

ES is currently the main data store for zagat.com which ends up being a sink from a data pipeline and more or less being used as Key-Value store on the query side. It has worked OK for our current use case, but definitely came with some pain points. Primary key fetches were way too slow for what we needed especially with some in-memory joins happening and we ended up sticking a cache in front of ES to satisfy our per…

If it's a sink from a data pipeline then presumably it's not your primary point-of-truth data store because if Elasticsearch gets corrupted you can rebuild it from the rest of the pipeline?

Yep, it's a bit more symbiotic than that unfortunately, but in general most of the data can be restored from an upstream source.

Re: Using Elasticsearch as the Primary Data Store in an ETL Pipeline

#43

Post by kn7 » Wed, Dec 05, 2018, 5:55 PM UTC

luord wrote: post #40

Would love to do/see similar benchmarks on Postgres using more performant setups. Like someone else said, the disparity is downright shocking.

Please see my reply to "someone else" you mentioned. If there is anything else you think I might have mistaken to employ during benchmarks, I am all ears.

Re: Using Elasticsearch as the Primary Data Store in an ETL Pipeline

#44

Post by luord » Thu, Dec 06, 2018, 2:45 AM UTC

kn7 wrote: post #43

luord wrote: post #40

Would love to do/see similar benchmarks on Postgres using more performant setups. Like someone else said, the disparity is downright shocking.

Please see my reply to "someone else" you mentioned. If there is anything else you think I might have mistaken to employ during benchmarks, I am all ears.

Yeah, I saw your reply. To be honest, I was mostly fishing for the possibility of finding a dataset with which I could perform the test myself.