Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Airflow and the Future of Data Engineering: A Q&A

medium.com

Post reply on HN
Print view View original

31–40 of 95 posts

Re: Airflow and the Future of Data Engineering: A Q&A

#31

Post by classybull »

kozikow wrote: post #6

Airflow works well for "static" jobs, but I miss something like airflow for dynamic jobs. By dynamic, I mean something like "user sent us some new data to process, create a custom graph just for this data". I can create new airflow graph per each processing pipeline with new dag id every time, but airflow was not created for use case like this and it's not working well in such scenario.

How about a custom sensor class that never timesout and you define its poke method to listen for data and processes it? Granted, you'll have to make sure you allocate enough workers for that..

Re: Airflow and the Future of Data Engineering: A Q&A

#33

Post by jcalabro »

[Disclaimer: I work for Composable] My team and I are working on a project that I would consider a competitor to Airflow. I'm not overly familiar with Airflow, but Composable seems to be fit for a much wider variety of use cases.

In Composable's DAG execution engine, you can pull in data from various sources (SQL, NoSQL, csv, json, restful endpoints, etc.) into our common data format. You can then easily transform, orchestrate, or analyze your data using our built-in Modules (blocks) or you can easily write your own. You can then view your resulting data all within the webapp.

Reading the comments, it seems like Composable supports a lot of the things people are asking for here that Airflow is lacking. Maybe check us out and let us know what you think!

For more information: Composable Site - https://composableanalytics.com/ Try it yourself - https://cloud.composableanalytics.com/ Composable's Blog - http://blog.composable.ai/

Re: Airflow and the Future of Data Engineering: A Q&A

#34

Post by erikbern »

nl wrote: post #2

Airflow. This framework is used by numerous companies and several of the biggest unicorns — Spotify, Lyft, Airbnb, Stripe, and others to power data engineering at massive scale. Is that correct? I've been using (and enjoying) Luigi[1] which came out of Spotify. I haven't seen anything about them switching to Airflow. Edit: Now I see in the interview there is this: About Luigi, it is simpler in scope than Airflow, and…

I'm one of the original Luigi authors, and used to maintain it for a while, merging PR's daily etc. I left Spotify a couple of years ago. Since then it's been maintained mostly by Arash Rouhani, who is increasingly getting busy with other things.

But it's very much in active development and there are multiple pull requests merged every day.

I haven't had a lot of time to check out Airflow, but it seems great. Data engineering and thinking of data processing as functional pipelines is a great paradigm, think we're going to see a lot of future development in this area. Luigi will probably evolve a lot over the next few years. Eventually I think there will be better frameworks. No idea if Airflow is a step change, I think there are still projects yet to be built that unifies everything beautifully

Re: Airflow and the Future of Data Engineering: A Q&A

#35

Post by j_s »

jcalabro wrote: post #33

[Disclaimer: I work for Composable] My team and I are working on a project that I would consider a competitor to Airflow. I'm not overly familiar with Airflow, but Composable seems to be fit for a much wider variety of use cases. In Composable's DAG execution engine, you can pull in data from various sources (SQL, NoSQL, csv, json, restful endpoints, etc.) into our common data format. You can then easily transform, o…

Is any aspect of your product open source? Doesn't look like it; this makes it a bit of an apples-vs-apple-pie comparison.

Re: Airflow and the Future of Data Engineering: A Q&A

#37

Post by batbomb »

jcalabro wrote: post #33

[Disclaimer: I work for Composable] My team and I are working on a project that I would consider a competitor to Airflow. I'm not overly familiar with Airflow, but Composable seems to be fit for a much wider variety of use cases. In Composable's DAG execution engine, you can pull in data from various sources (SQL, NoSQL, csv, json, restful endpoints, etc.) into our common data format. You can then easily transform, o…

It sounds like you are more of a Beam/dataflow competitor than an airflow/workflow competitor.

Re: Airflow and the Future of Data Engineering: A Q&A

#39

Post by gdulli »

gregn610 wrote: post #36

How relevant are Airflow and similar to those of us who aren't operating at unicorn scale but are shuffling hundreds of CSVs & Excels and wrangling RDBMS with SQL?

Airflow is more about scaling your number of processes, not the size of your data. It's relevant if you want to have a higher-level system for managing processes independent of what those processes are.

Re: Airflow and the Future of Data Engineering: A Q&A

#40

Post by gallamine »

sandGorgon wrote: post #27

MHO There seems to be quite some conceptual overlap between Airflow's DAG and Spark RDD. It seems to me that Airflow is Spark-on-a-db ... or rather Spark is Airflow-on-Hadoop. Does anyone know what the difference is ?

Airflow doesn't have anything to do with data storage, movement or processing. It's a way to chain commands together in such a way so that you can define "do Z after Y a Z finish", for example. Many people use it like a nice version of cron with a UI, alerting, and retries.
Post reply on HN