Airflow and the Future of Data Engineering: A Q&A
1–10 of 95 posts
Re: Airflow and the Future of Data Engineering: A Q&A
#2Airflow. This framework is used by numerous companies and several of the biggest unicorns — Spotify, Lyft, Airbnb, Stripe, and others to power data engineering at massive scale.
Is that correct? I've been using (and enjoying) Luigi[1] which came out of Spotify. I haven't seen anything about them switching to Airflow.
Edit: Now I see in the interview there is this:
About Luigi, it is simpler in scope than Airflow, and perhaps we’re more complementary than competition. From what I gather, the main maintainer of the product has left Spotify and apparently they are now using Airflow internally for [at least] some of their use cases. I do not have the full story here and would like to hear more about it. I’m thinking that many of the companies choosing Luigi today might also choose Airflow later as they develop the need for the extra set of features that Airflow offers.
But there are 2 day old commits in the Luigi directory, so I don't know. I like Airflow too, but it did seems a lot more complicated the Luigi when I played with it.
Re: Airflow and the Future of Data Engineering: A Q&A
#3+1 for Airflow. I use it every day to handle tasks with many components and dependencies. I love that everything is code & version-controlled.
I do wish it had a REST API though.
Re: Airflow and the Future of Data Engineering: A Q&A
#4+1 for Airflow. I use it every day to handle tasks with many components and dependencies. I love that everything is code & version-controlled. I do wish it had a REST API though.
You can always expose the REST API. Its pretty easy considering they are just flask blueprints. Since, you can make your own custom plugin - You can build a lot using existing infrastructure that Airflow provides.
Re: Airflow and the Future of Data Engineering: A Q&A
#5Amazing framework with a lot of functionalities. A tool built to be extensible just what an open source software should look like :)
Re: Airflow and the Future of Data Engineering: A Q&A
#6Airflow works well for "static" jobs, but I miss something like airflow for dynamic jobs.
By dynamic, I mean something like "user sent us some new data to process, create a custom graph just for this data". I can create new airflow graph per each processing pipeline with new dag id every time, but airflow was not created for use case like this and it's not working well in such scenario.
Re: Airflow and the Future of Data Engineering: A Q&A
#7Airflow works well for "static" jobs, but I miss something like airflow for dynamic jobs. By dynamic, I mean something like "user sent us some new data to process, create a custom graph just for this data". I can create new airflow graph per each processing pipeline with new dag id every time, but airflow was not created for use case like this and it's not working well in such scenario.
[deleted]
Re: Airflow and the Future of Data Engineering: A Q&A
#8Airflow works well for "static" jobs, but I miss something like airflow for dynamic jobs. By dynamic, I mean something like "user sent us some new data to process, create a custom graph just for this data". I can create new airflow graph per each processing pipeline with new dag id every time, but airflow was not created for use case like this and it's not working well in such scenario.
Airflow is just the workflow management layer on top of your data pipeline. The flexibility to generate custom graphs based on user-specific parameters should be handled within a pipeline task.
Based on your example, I would have a single dag that would 1. get user data and 2. generate a graph.
All the flexibility should be defined in whatever function, script or program you define to generate the graph.
Re: Airflow and the Future of Data Engineering: A Q&A
#9Airflow works well for "static" jobs, but I miss something like airflow for dynamic jobs. By dynamic, I mean something like "user sent us some new data to process, create a custom graph just for this data". I can create new airflow graph per each processing pipeline with new dag id every time, but airflow was not created for use case like this and it's not working well in such scenario.
[deleted]
Re: Airflow and the Future of Data Engineering: A Q&A
#10Airflow. This framework is used by numerous companies and several of the biggest unicorns — Spotify, Lyft, Airbnb, Stripe, and others to power data engineering at massive scale. Is that correct? I've been using (and enjoying) Luigi[1] which came out of Spotify. I haven't seen anything about them switching to Airflow. Edit: Now I see in the interview there is this: About Luigi, it is simpler in scope than Airflow, and…
They've been listed as contributors on the Apache Airflow repo[0]
[0] https://github.com/apache/incubator-airflow/blame/master/REA...