Using Elasticsearch as the Primary Data Store in an ETL Pipeline
1–10 of 44 posts
Re: Using Elasticsearch as the Primary Data Store in an ETL Pipeline
#2... If your primary data store should be Elasticsearch. Nice write up of everything, though, for sure.
Re: Using Elasticsearch as the Primary Data Store in an ETL Pipeline
#3> "Enough blaming the former engineer."
I was one of them. I don't work there anymore.
I believe is it not the actual situation in bol.com. If it is, I would be disappointed.
Last I remember, Bol.com has really good set of ops and dev tooling on hadoop, hbase, spark, flink etc. for scheduling, running jobs etc.
I wouldn't know why they replicated data both on hbase, elastic search etc. Having read the blog, I don't see how this fits the event sourcing pattern that bol.com was trying to implement and also, the idea of self service BI that they envisioned.
Re: Using Elasticsearch as the Primary Data Store in an ETL Pipeline
#4I'm certain I'm missing something very obvious about Elasticsearch and other NoSQL data stores.
But in my brief experience with an Elasticsearch-backed web application I found it difficult to write integration tests for the portions of code that dealt with ES. Thanks to ES's "eventually consistent" nature, tests would fail intermittently because we'd be e.g. querying some data that was written to ES but hadn't been fully persisted yet. ES gives you some "flush the data to disk right now, please" commands but they're merely suggestions and cannot be relied upon.
Obviously, you want to stub/mock out as many of those actual physical database reads/writes as possible, but sometimes what you want to test is the ES queries themselves and I don't know what on earth the best practice is there.
Just to reiterate, this is an issue that any non-ACID datastore would experience. I'm not criticizing them, I'm just sort of wondering how people typically solve that...
Re: Using Elasticsearch as the Primary Data Store in an ETL Pipeline
#5> "Enough blaming the former engineer." I was one of them. I don't work there anymore. I believe is it not the actual situation in bol.com. If it is, I would be disappointed. Last I remember, Bol.com has really good set of ops and dev tooling on hadoop, hbase, spark, flink etc. for scheduling, running jobs etc. I wouldn't know why they replicated data both on hbase, elastic search etc. Having read the blog, I don't s…
Any feedback on how this went? The even sourcing pattern I mean.
Re: Using Elasticsearch as the Primary Data Store in an ETL Pipeline
#6I'm certain I'm missing something very obvious about Elasticsearch and other NoSQL data stores. But in my brief experience with an Elasticsearch-backed web application I found it difficult to write integration tests for the portions of code that dealt with ES. Thanks to ES's "eventually consistent" nature, tests would fail intermittently because we'd be e.g. querying some data that was written to ES but hadn't been f…
That is indeed the case and we are also bitten by that. The most effective work around we managed to find is to flush periodically (say every 300ms) for a certain timeout period before reading from ES again for checks. Though even then, ITs still fail time to time.
Re: Using Elasticsearch as the Primary Data Store in an ETL Pipeline
#7I'm certain I'm missing something very obvious about Elasticsearch and other NoSQL data stores. But in my brief experience with an Elasticsearch-backed web application I found it difficult to write integration tests for the portions of code that dealt with ES. Thanks to ES's "eventually consistent" nature, tests would fail intermittently because we'd be e.g. querying some data that was written to ES but hadn't been f…
I use `refresh=True` on my insert/update/delete, which forces the writes to complete. Then all the reads work as you would expect.
Re: Using Elasticsearch as the Primary Data Store in an ETL Pipeline
#8[deleted]
Re: Using Elasticsearch as the Primary Data Store in an ETL Pipeline
#9> "Enough blaming the former engineer." I was one of them. I don't work there anymore. I believe is it not the actual situation in bol.com. If it is, I would be disappointed. Last I remember, Bol.com has really good set of ops and dev tooling on hadoop, hbase, spark, flink etc. for scheduling, running jobs etc. I wouldn't know why they replicated data both on hbase, elastic search etc. Having read the blog, I don't s…
Hey Debarsh! First, thanks for taking time to read such a lengthy post.
If I am not mistaken the majority of the PL/SQL glue is owned by Gert, though you might recall better. Quite some VCS history was lost while migrating from SVN to Git. ;-)
The reason we are "replicating" the entire data is to 1) determine the affected products and 2) re-execute the relevant configurations (facets, synonyms, etc.) while making retroactive changes. (For instance, say someone has changed the PL/SQL of "leeftijd" facet.) Here, the storage is required to allow querying on every field, for (1), and on id, for (2). While id-based bulk querying is (almost) supported by every ETL source, querying on every field is not. Hence, we "replicate" the sources on our side to suffice these needs. Actually, the entire point of the post was to explain this problem, but apparently it was not clear enough.
For your remarks on event sourcing and BI, I am a little bit puzzled. I will need some elaboration on these remarks. We do have event sourcing on our side (that is how we can replay in case of need) and BI is not really interested in ETL data. Maybe I misunderstood you?
I am also confused by how you relate scheduling/running PL/SQL jobs via Hadoop, Spark, Flink, etc. Did you see the link to Redwood Explorer I shared in the post?
Re: Using Elasticsearch as the Primary Data Store in an ETL Pipeline
#10I'm certain I'm missing something very obvious about Elasticsearch and other NoSQL data stores. But in my brief experience with an Elasticsearch-backed web application I found it difficult to write integration tests for the portions of code that dealt with ES. Thanks to ES's "eventually consistent" nature, tests would fail intermittently because we'd be e.g. querying some data that was written to ES but hadn't been f…
Elasticsearch has a "refresh" query flag to force immediate consistency: https://www.elastic.co/guide/en/elasticsearch/reference/curr...