Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Using Elasticsearch as the Primary Data Store in an ETL Pipeline

vlkan.com

Post reply on HN
Print view View original

31–40 of 44 posts

Re: Using Elasticsearch as the Primary Data Store in an ETL Pipeline

#31

Post by debarshri »

kn7 wrote: post #9
debarshri wrote: post #3

> "Enough blaming the former engineer." I was one of them. I don't work there anymore. I believe is it not the actual situation in bol.com. If it is, I would be disappointed. Last I remember, Bol.com has really good set of ops and dev tooling on hadoop, hbase, spark, flink etc. for scheduling, running jobs etc. I wouldn't know why they replicated data both on hbase, elastic search etc. Having read the blog, I don't s…

Hey Debarsh! First, thanks for taking time to read such a lengthy post. If I am not mistaken the majority of the PL/SQL glue is owned by Gert, though you might recall better. Quite some VCS history was lost while migrating from SVN to Git. ;-) The reason we are "replicating" the entire data is to 1) determine the affected products and 2) re-execute the relevant configurations (facets, synonyms, etc.) while making ret…

> BI is not really interested in ETL data

Isn't ETL an intermediary in BI? I think I am a bit confused, to give some context, this is my understanding, you have all the services generating data, you have ETL jobs, that extract data from these services, transform and move the data to a star or snowflake schema in RDBMS prepared for BI tools for query efficiently.

Re: Using Elasticsearch as the Primary Data Store in an ETL Pipeline

#32

Post by simonw »

buckhx wrote: post #18

ES is currently the main data store for zagat.com which ends up being a sink from a data pipeline and more or less being used as Key-Value store on the query side. It has worked OK for our current use case, but definitely came with some pain points. Primary key fetches were way too slow for what we needed especially with some in-memory joins happening and we ended up sticking a cache in front of ES to satisfy our per…

If it's a sink from a data pipeline then presumably it's not your primary point-of-truth data store because if Elasticsearch gets corrupted you can rebuild it from the rest of the pipeline?

Re: Using Elasticsearch as the Primary Data Store in an ETL Pipeline

#34

Post by simonw »

If Elasticsearch is at the end of an ETL pipeline, does that mean that if Elasticsearch gets corrupted you can rebuild it by re-running the pipeline?

If so I wouldn't call this a "primary data store", since durability isn't critical.

The article says:

> After drafting many blueprints, we went for a Java service backed by Elasticsearch as the primary storage! This idea brought shivers to even the most senior Elasticsearch consultants hired

I'll shiver if Elasticsearch corrupting irreversibly loses data, but if it can be rebuilt from another source I don't see any problems with it at all.

Re: Using Elasticsearch as the Primary Data Store in an ETL Pipeline

#35

Post by KennyCason »

simonw wrote: post #34

If Elasticsearch is at the end of an ETL pipeline, does that mean that if Elasticsearch gets corrupted you can rebuild it by re-running the pipeline? If so I wouldn't call this a "primary data store", since durability isn't critical. The article says: > After drafting many blueprints, we went for a Java service backed by Elasticsearch as the primary storage! This idea brought shivers to even the most senior Elasticse…

Agreed.

We’ve been running large Elasticsearch clusters as our primary search/analytics engine. While it’s overall very stable, stuff does occasionally happen that requires an index rebuild. We use HBase as our primary store and index via map/reduce or Spark Batch.

As much as I love Elasticsearch, I definitely wouldn’t be able to sleep at night knowing it was the primary datastore.

Re: Using Elasticsearch as the Primary Data Store in an ETL Pipeline

#36

Post by kn7 »

billoneil wrote: post #26

> The reason PostgreSQL results were omitted is no matter what kind of optimization we throw at it, the benchmark always took more than 2 hours, regardless of partitioning, whereas MongoDB and Elasticsearch took a couple of minutes. This seems like something was done incorrectly the comparison shouldn't be that drastic. > Just one PostgreSQL 9.6.10 instance (shared_buffers = 128MB) This looks way too low. The postgre…

It was indeed an incorrect snippet -- removed it. We do have a group of PostgreSQL experts within the company and we let them tune the database for the benchmark. Let me remind, this was not a tune once, run once operation. We spent close to a month to make sure that we are not missing anything obvious for each storage engine. But as I explained, nothing much worked in case of PostgreSQL.

Re: Using Elasticsearch as the Primary Data Store in an ETL Pipeline

#37

Post by kn7 »

skullborg wrote: post #28
billoneil wrote: post #26

> The reason PostgreSQL results were omitted is no matter what kind of optimization we throw at it, the benchmark always took more than 2 hours, regardless of partitioning, whereas MongoDB and Elasticsearch took a couple of minutes. This seems like something was done incorrectly the comparison shouldn't be that drastic. > Just one PostgreSQL 9.6.10 instance (shared_buffers = 128MB) This looks way too low. The postgre…

They set out on a 4 year journey to improve their ETL, but didn't take 1 second to change a conservative global config default. Can barely take the rest of the article seriously after a blunder like that

See my response above, it was indeed a typo from my side. I am sorry to hear that it spoiled the rest of the post for you.

Re: Using Elasticsearch as the Primary Data Store in an ETL Pipeline

#38

Post by kn7 »

simonw wrote: post #34

If Elasticsearch is at the end of an ETL pipeline, does that mean that if Elasticsearch gets corrupted you can rebuild it by re-running the pipeline? If so I wouldn't call this a "primary data store", since durability isn't critical. The article says: > After drafting many blueprints, we went for a Java service backed by Elasticsearch as the primary storage! This idea brought shivers to even the most senior Elasticse…

We store the real-time content stream in a separate bulk storage unit (e.g., BigQuery) with a certain retention window, but the ETL'ed documents are always on ES. Given a plain event (i.e., not ETL'ed document) is not much of a value for search, I would not call the stream storage as the primary storage. It just assists us to re-build the ETL state in case of an emergency.

Re: Using Elasticsearch as the Primary Data Store in an ETL Pipeline

#39

Post by billoneil »

kn7 wrote: post #36
billoneil wrote: post #26

> The reason PostgreSQL results were omitted is no matter what kind of optimization we throw at it, the benchmark always took more than 2 hours, regardless of partitioning, whereas MongoDB and Elasticsearch took a couple of minutes. This seems like something was done incorrectly the comparison shouldn't be that drastic. > Just one PostgreSQL 9.6.10 instance (shared_buffers = 128MB) This looks way too low. The postgre…

It was indeed an incorrect snippet -- removed it. We do have a group of PostgreSQL experts within the company and we let them tune the database for the benchmark. Let me remind, this was not a tune once, run once operation. We spent close to a month to make sure that we are not missing anything obvious for each storage engine. But as I explained, nothing much worked in case of PostgreSQL.

Either way, something still seems off. If MongoDB and Elasticsearch were 3-5x more performant I would still find that surprising. A > 10-20x difference really seems like a configuration or implementation issue.
Post reply on HN