Using Elasticsearch as the Primary Data Store in an ETL Pipeline • Hacker News Board

Re: Using Elasticsearch as the Primary Data Store in an ETL Pipeline

#11

Post by athenot » Fri, Nov 30, 2018, 7:39 PM UTC

JohnBooty wrote: post #4

I'm certain I'm missing something very obvious about Elasticsearch and other NoSQL data stores. But in my brief experience with an Elasticsearch-backed web application I found it difficult to write integration tests for the portions of code that dealt with ES. Thanks to ES's "eventually consistent" nature, tests would fail intermittently because we'd be e.g. querying some data that was written to ES but hadn't been f…

ES comes in handy with very large sets of data. If it fits on one (or a handful) of nodes, you're probably better served by a relational database.

When you try to scale something to large sizes AND want high availability, it's pretty much a given you'll be dealing with eventual consistency.

We use ES to ingest billions of records per day. For us, being able to immediately query a row that was just added is less important than being able to deal with the volume in relatively predictable performances.

Re: Using Elasticsearch as the Primary Data Store in an ETL Pipeline

#12

Post by barbecue_sauce » Fri, Nov 30, 2018, 7:40 PM UTC

kn7 wrote: post #9

debarshri wrote: post #3

> "Enough blaming the former engineer." I was one of them. I don't work there anymore. I believe is it not the actual situation in bol.com. If it is, I would be disappointed. Last I remember, Bol.com has really good set of ops and dev tooling on hadoop, hbase, spark, flink etc. for scheduling, running jobs etc. I wouldn't know why they replicated data both on hbase, elastic search etc. Having read the blog, I don't s…

Hey Debarsh! First, thanks for taking time to read such a lengthy post. If I am not mistaken the majority of the PL/SQL glue is owned by Gert, though you might recall better. Quite some VCS history was lost while migrating from SVN to Git. ;-) The reason we are "replicating" the entire data is to 1) determine the affected products and 2) re-execute the relevant configurations (facets, synonyms, etc.) while making ret…

I am not Debarsh, and I am not a data engineer, but isn't the purpose of ETL for transforming data into a more accessible/palatable form for BI?

Re: Using Elasticsearch as the Primary Data Store in an ETL Pipeline

#13

Post by kn7 » Fri, Nov 30, 2018, 7:46 PM UTC

barbecue_sauce wrote: post #12

kn7 wrote: post #9

Earlier quoted context omitted.

Hey Debarsh! First, thanks for taking time to read such a lengthy post. If I am not mistaken the majority of the PL/SQL glue is owned by Gert, though you might recall better. Quite some VCS history was lost while migrating from SVN to Git. ;-) The reason we are "replicating" the entire data is to 1) determine the affected products and 2) re-execute the relevant configurations (facets, synonyms, etc.) while making ret…

I am not Debarsh, and I am not a data engineer, but isn't the purpose of ETL for transforming data into a more accessible/palatable form for BI?

Bol has plenty of other ETL pipelines for BI. What I meant is the data cooked for search is not (much) of interest to BI, yet. Though we do have other means to feed BI for search-relevant content.

Re: Using Elasticsearch as the Primary Data Store in an ETL Pipeline

#14

Post by kn7 » Fri, Nov 30, 2018, 7:48 PM UTC

jacobr1 wrote: post #7

JohnBooty wrote: post #4

I'm certain I'm missing something very obvious about Elasticsearch and other NoSQL data stores. But in my brief experience with an Elasticsearch-backed web application I found it difficult to write integration tests for the portions of code that dealt with ES. Thanks to ES's "eventually consistent" nature, tests would fail intermittently because we'd be e.g. querying some data that was written to ES but hadn't been f…

I use `refresh=True` on my insert/update/delete, which forces the writes to complete. Then all the reads work as you would expect.

That still doesn't guarantee that a consecutive read is gonna get the last state.

Re: Using Elasticsearch as the Primary Data Store in an ETL Pipeline

#15

Post by kn7 » Fri, Nov 30, 2018, 7:53 PM UTC

manigandham wrote: post #10

JohnBooty wrote: post #4

I'm certain I'm missing something very obvious about Elasticsearch and other NoSQL data stores. But in my brief experience with an Elasticsearch-backed web application I found it difficult to write integration tests for the portions of code that dealt with ES. Thanks to ES's "eventually consistent" nature, tests would fail intermittently because we'd be e.g. querying some data that was written to ES but hadn't been f…

Elasticsearch has a "refresh" query flag to force immediate consistency: https://www.elastic.co/guide/en/elasticsearch/reference/curr...

That still doesn't guarantee that a consecutive read is gonna get the last state. Welcome to the wonderful world of "eventual consistency".

Re: Using Elasticsearch as the Primary Data Store in an ETL Pipeline

#16

Post by haggy » Fri, Nov 30, 2018, 8:08 PM UTC

kn7 wrote: post #14

jacobr1 wrote: post #7

Earlier quoted context omitted.

I use `refresh=True` on my insert/update/delete, which forces the writes to complete. Then all the reads work as you would expect.

That still doesn't guarantee that a consecutive read is gonna get the last state.

Direct from ES documentation on the `refresh` flag: "Refresh the relevant primary and replica shards (not the whole index) immediately after the operation occurs, so that the updated document appears in search results immediately."

Do you have other information about the refresh flag because their documentation clearly states that forcing a refresh is applied to primary and replica shards meaning that it will be available for query directly after the call to refresh is made.

Re: Using Elasticsearch as the Primary Data Store in an ETL Pipeline

#17

Post by bobbyi_settv » Fri, Nov 30, 2018, 8:21 PM UTC

kn7 wrote: post #14

jacobr1 wrote: post #7

Earlier quoted context omitted.

I use `refresh=True` on my insert/update/delete, which forces the writes to complete. Then all the reads work as you would expect.

That still doesn't guarantee that a consecutive read is gonna get the last state.

Yes it does. I'm not sure what you think the refresh operation is if not that.

I've run countless integration tests with ES and never seen something fail due to refresh not working as advertised. If you have, what version of ES was it? Can you give some sample code that sporadically exhibits the problem?

Re: Using Elasticsearch as the Primary Data Store in an ETL Pipeline

#18

Post by buckhx » Fri, Nov 30, 2018, 8:24 PM UTC

ES is currently the main data store for zagat.com which ends up being a sink from a data pipeline and more or less being used as Key-Value store on the query side. It has worked OK for our current use case, but definitely came with some pain points. Primary key fetches were way too slow for what we needed especially with some in-memory joins happening and we ended up sticking a cache in front of ES to satisfy our performance reqs.

We had a tight deadline on implementation (3 months to extract from Google) and chose ES in order to satisfy a kv store as well as TF-IDF corpus search.

Re: Using Elasticsearch as the Primary Data Store in an ETL Pipeline

#19

Post by manigandham » Fri, Nov 30, 2018, 8:26 PM UTC

kn7 wrote: post #15

manigandham wrote: post #10

Earlier quoted context omitted.

Elasticsearch has a "refresh" query flag to force immediate consistency: https://www.elastic.co/guide/en/elasticsearch/reference/curr...

That still doesn't guarantee that a consecutive read is gonna get the last state. Welcome to the wonderful world of "eventual consistency".

How so? That's the entire point of the feature. What else is there to update?

Re: Using Elasticsearch as the Primary Data Store in an ETL Pipeline

#20

Post by manigandham » Fri, Nov 30, 2018, 8:30 PM UTC

athenot wrote: post #11

JohnBooty wrote: post #4

I'm certain I'm missing something very obvious about Elasticsearch and other NoSQL data stores. But in my brief experience with an Elasticsearch-backed web application I found it difficult to write integration tests for the portions of code that dealt with ES. Thanks to ES's "eventually consistent" nature, tests would fail intermittently because we'd be e.g. querying some data that was written to ES but hadn't been f…

ES comes in handy with very large sets of data. If it fits on one (or a handful) of nodes, you're probably better served by a relational database. When you try to scale something to large sizes AND want high availability, it's pretty much a given you'll be dealing with eventual consistency. We use ES to ingest billions of records per day. For us, being able to immediately query a row that was just added is less impor…

It's not a given. Plenty of distributed databases support strong consistency.

ES is not meant to be an OLTP database. It's a search index with a much better wrapper around Lucene, but the distributed part has always been weak. The last several years of updates have primarily been around fixing the home-grown replication and storage.