Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

How we run Spark and Sqoop in production

thumbtack.com

Post reply on HN
Print view View original

11–19 of 19 posts

Re: How we run Spark and Sqoop in production

#11

Post by machbio »

rathboma wrote: post #6
ktamura wrote: post #2

Any good alternatives for Sqoop? I feel that an ETL tool just for HDFS is too limiting and leads to further fragmentation on the data pipeline.

Using Sqoop from something like Luigi as the ETL manager is a pretty great workflow - https://github.com/spotify/luigi You can define dependencies between jobs based on output file which allows you to re-run only part of your pipeline

Thats a great idea - but could you elaborate on the scheduling of jobs on Luigi - it does not have a scheduler like AirFlow - how do you schedule Luigi tasks ?

Re: How we run Spark and Sqoop in production

#12

Post by dn5 »

ktamura wrote: post #2

Any good alternatives for Sqoop? I feel that an ETL tool just for HDFS is too limiting and leads to further fragmentation on the data pipeline.

Have a look at Kafka Connect (http://docs.confluent.io/2.0.0/connect). The JDBC Connector will poll for database changes changes and push them to a Kafka topic. Means you should see all the changes in the database rather than a snapshot say once a day

Re: How we run Spark and Sqoop in production

#13

Post by rathboma »

machbio wrote: post #11
rathboma wrote: post #6

Earlier quoted context omitted.

Using Sqoop from something like Luigi as the ETL manager is a pretty great workflow - https://github.com/spotify/luigi You can define dependencies between jobs based on output file which allows you to re-run only part of your pipeline

Thats a great idea - but could you elaborate on the scheduling of jobs on Luigi - it does not have a scheduler like AirFlow - how do you schedule Luigi tasks ?

Check out this Foursquare talk that goes through how we used to do scheduling -- basically you make jobs dependent on a date - http://www.slideshare.net/OpenAnayticsMeetup/luigi-presentat...

Re: How we run Spark and Sqoop in production

#15

Post by natekupp »

rahij wrote: post #14
natekupp wrote: post #9

hey all, feel free to reach out to me either on this thread, or directly at nate[at]thumbtack.com if I can answer any questions!

I had to connect to a US VPN to access the jobs page. Is that intentional?

thanks for flagging, I'll look into it!

Re: How we run Spark and Sqoop in production

#16

Post by allengeorge »

machbio wrote: post #11
rathboma wrote: post #6

Earlier quoted context omitted.

Using Sqoop from something like Luigi as the ETL manager is a pretty great workflow - https://github.com/spotify/luigi You can define dependencies between jobs based on output file which allows you to re-run only part of your pipeline

Thats a great idea - but could you elaborate on the scheduling of jobs on Luigi - it does not have a scheduler like AirFlow - how do you schedule Luigi tasks ?

You have to use an external scheduler. We built one on top of AP Scheduler: https://apscheduler.readthedocs.io/en/latest/
Post reply on HN