How we run Spark and Sqoop in production
thumbtack.com
How we run Spark and Sqoop in production
1–10 of 19 posts
Re: How we run Spark and Sqoop in production
#2Any good alternatives for Sqoop? I feel that an ETL tool just for HDFS is too limiting and leads to further fragmentation on the data pipeline.
Re: How we run Spark and Sqoop in production
#3Any good alternatives for Sqoop? I feel that an ETL tool just for HDFS is too limiting and leads to further fragmentation on the data pipeline.
We've been trying out goldengate to get streaming replication, but it has proven rather unreliable. Stops replicating if you sneeze in its general vicinity. I wonder whether the alternatives like shareplex and tungsten are more reliable.
Re: How we run Spark and Sqoop in production
#4Any good alternatives for Sqoop? I feel that an ETL tool just for HDFS is too limiting and leads to further fragmentation on the data pipeline.
We've been trying out goldengate to get streaming replication, but it has proven rather unreliable. Stops replicating if you sneeze in its general vicinity. I wonder whether the alternatives like shareplex and tungsten are more reliable.
You can try out Striim for streaming data integration (full disclosure, I work there):
Re: How we run Spark and Sqoop in production
#5Any good alternatives for Sqoop? I feel that an ETL tool just for HDFS is too limiting and leads to further fragmentation on the data pipeline.
I feel like I've recommended it enough times I'm turning into a shill but Pentaho is an open-source and commercially supported ETL tool that will natively do what you want, or call sqoop when you discover that's kinda slow. :-) And no I definitely don't work there.
Re: How we run Spark and Sqoop in production
#6Any good alternatives for Sqoop? I feel that an ETL tool just for HDFS is too limiting and leads to further fragmentation on the data pipeline.
Using Sqoop from something like Luigi as the ETL manager is a pretty great workflow - https://github.com/spotify/luigi
You can define dependencies between jobs based on output file which allows you to re-run only part of your pipeline
Re: How we run Spark and Sqoop in production
#7Thumbtack is great for lazy consumers. But word on the street is they contribute to too much price pressure on the market, therefore drive the overall quality of services down. Good work Thumbtack! You have figured out a convenient way to sacrifice long term value in favor of short term profit.
Re: How we run Spark and Sqoop in production
#8Any good alternatives for Sqoop? I feel that an ETL tool just for HDFS is too limiting and leads to further fragmentation on the data pipeline.
Using Sqoop from something like Luigi as the ETL manager is a pretty great workflow - https://github.com/spotify/luigi You can define dependencies between jobs based on output file which allows you to re-run only part of your pipeline
+1 to this, we kick off our Sqoop jobs using Airflow - http://airbnb.io/projects/airflow/
Airflow is very similar to Luigi; we've been using in in production to schedule all of our workflows for ~4 months now and it's worked out really well for us.
Re: How we run Spark and Sqoop in production
#9hey all, feel free to reach out to me either on this thread, or directly at nate[at]thumbtack.com if I can answer any questions!
Re: How we run Spark and Sqoop in production
#10Any good alternatives for Sqoop? I feel that an ETL tool just for HDFS is too limiting and leads to further fragmentation on the data pipeline.
Ever since using Apache Spark's Data Sources API was released, I have been relying no different Spark Data Source packages for my ETL jobs.