Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Airflow and the Future of Data Engineering: A Q&A

medium.com

Post reply on HN
Print view View original

21–30 of 95 posts

Re: Airflow and the Future of Data Engineering: A Q&A

#21

Post by thibaut_barrere »

kozikow wrote: post #6

Airflow works well for "static" jobs, but I miss something like airflow for dynamic jobs. By dynamic, I mean something like "user sent us some new data to process, create a custom graph just for this data". I can create new airflow graph per each processing pipeline with new dag id every time, but airflow was not created for use case like this and it's not working well in such scenario.

If you are also using Ruby, you can use Kiba ETL (disclaimer: I'm Kiba ETL author) in conjunction with Sidekiq (quick example: https://gist.github.com/thbar/ec90bbd5877d1aae40510fadd43206...).

This allows to build ETL jobs which react to all sorts of external application triggers (such as an upload made by a user in that case, but it could be an API notification / webhook etc).

Re: Airflow and the Future of Data Engineering: A Q&A

#22

Post by nl »

kozikow wrote: post #6

Airflow works well for "static" jobs, but I miss something like airflow for dynamic jobs. By dynamic, I mean something like "user sent us some new data to process, create a custom graph just for this data". I can create new airflow graph per each processing pipeline with new dag id every time, but airflow was not created for use case like this and it's not working well in such scenario.

This is one part of Luigi that I haven't tried yet, but https://luigi.readthedocs.io/en/stable/tasks.html#dynamic-de... might help.

Re: Airflow and the Future of Data Engineering: A Q&A

#24

Post by smooc »

concam wrote: post #13
chishaku wrote: post #8

Earlier quoted context omitted.

Airflow is just the workflow management layer on top of your data pipeline. The flexibility to generate custom graphs based on user-specific parameters should be handled within a pipeline task. Based on your example, I would have a single dag that would 1. get user data and 2. generate a graph. All the flexibility should be defined in whatever function, script or program you define to generate the graph.

I think they are referring to "event-driven" DAGs which would be both shaped and triggered dynamically. You can accomplish this now but it feels a bit hacky and is pretty clear that it goes against the Airflow paradigm of static, slowly-changing workflows

Starting from 1.8 you will be able to trigger dags through a rest API, that is fully supported.

Shaping DAGs dynamically poses a challenge to the scheduler on how to 'predict' what tasks need to run in the future. The scheduler needs to evaluate which tasks will need to run, without actually executing these tasks themselves. For Airflow in its current state that is a chicken and egg problem.

For the future, I can think of allowing dynamic dags being described through the Rest API, but that is definitely further out and has not really popped up yet on the horizon.

Re: Airflow and the Future of Data Engineering: A Q&A

#25

Post by ktamura »

nl wrote: post #2

Airflow. This framework is used by numerous companies and several of the biggest unicorns — Spotify, Lyft, Airbnb, Stripe, and others to power data engineering at massive scale. Is that correct? I've been using (and enjoying) Luigi[1] which came out of Spotify. I haven't seen anything about them switching to Airflow. Edit: Now I see in the interview there is this: About Luigi, it is simpler in scope than Airflow, and…

If simplicity and non-Python-centricity matter, I encourage folks to look into Digdag [1][2].

It's Ansible for Workflow Management.

While both Luigi and Airflow (somewhat rightfully) assume the user to know/have affinity for Python, Digdag focuses on ease of use and helping enterprises move data around many systems.

If we learned one thing from today's S3 outage, it's not enough to use multiple cloud infrastructure providers: you should probably have your data in multiple cloud providers as well.

[1] https://www.digdag.io

[2] https://github.com/treasure-data/digdag

Re: Airflow and the Future of Data Engineering: A Q&A

#26

Post by throwaway_374 »

Minor gripe - why can I not execute an entire DAG (end to end) from the UI? Also trying to execute single tasks from the UI using the "run" functionality gives a CeleryExecutor requirement error... sorry, I know this isn't the help forums but it sounds like the most trivial tasks were overlooked.

Re: Airflow and the Future of Data Engineering: A Q&A

#28

Post by ubolonton_ »

ktamura wrote: post #25
nl wrote: post #2

Airflow. This framework is used by numerous companies and several of the biggest unicorns — Spotify, Lyft, Airbnb, Stripe, and others to power data engineering at massive scale. Is that correct? I've been using (and enjoying) Luigi[1] which came out of Spotify. I haven't seen anything about them switching to Airflow. Edit: Now I see in the interview there is this: About Luigi, it is simpler in scope than Airflow, and…

If simplicity and non-Python-centricity matter, I encourage folks to look into Digdag [1][2]. It's Ansible for Workflow Management. While both Luigi and Airflow (somewhat rightfully) assume the user to know/have affinity for Python, Digdag focuses on ease of use and helping enterprises move data around many systems. If we learned one thing from today's S3 outage, it's not enough to use multiple cloud infrastructure p…

Thanks! I skimmed through the doc. It looks very clean from a high-level view.

Re: Airflow and the Future of Data Engineering: A Q&A

#29

Post by classybull »

We're currently in a PoC phase of implementing Airflow and testing it out versus Luigi. So far, what I've liked, is that Airflow seems to be much more extensible and modular than Luigi. Getting Luigi to play nicely with our particular set of constraints was painful, and subclassing the Task was also tricky because it was way more opinionated about the structure of the class. Airflow seems way less so. There also seems to be way more right out of the gate in terms of built in task types. And the UI looks nicer.

We haven't made our final determination yet, but Airflow at the current moment feels better.

Re: Airflow and the Future of Data Engineering: A Q&A

#30

Post by classybull »

smooc wrote: post #18
cardosof wrote: post #14

I'm used to running R scripts with cron to handle some batch jobs and I'm no python dev. Would it be easy to start using Airflow?

DAGS in Airflow can just be a few lines. Some understanding of the syntax of python is required. But you can start simple and add complexity as you require it.

To add to that, you can create a DAG class that's really just a wrapper that executes your R script and waits for its return code.
Post reply on HN