Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Airflow and the Future of Data Engineering: A Q&A

medium.com

Post reply on HN
Print view View original

41–50 of 95 posts

Re: Airflow and the Future of Data Engineering: A Q&A

#41

Post by gdulli »

kozikow wrote: post #6

Airflow works well for "static" jobs, but I miss something like airflow for dynamic jobs. By dynamic, I mean something like "user sent us some new data to process, create a custom graph just for this data". I can create new airflow graph per each processing pipeline with new dag id every time, but airflow was not created for use case like this and it's not working well in such scenario.

I've started working on a code generator for Airflow. Not primarily because I needed dynamic jobs, but more because I didn't want to keep writing the Airflow boilerplate.

I imagine I'll eventually need to add some sort of management system to move these dynamic jobs in and out of Airflow to keep them from bloating the database or cluttering the UI.

Re: Airflow and the Future of Data Engineering: A Q&A

#42

Post by sandGorgon »

gallamine wrote: post #40
sandGorgon wrote: post #27

MHO There seems to be quite some conceptual overlap between Airflow's DAG and Spark RDD. It seems to me that Airflow is Spark-on-a-db ... or rather Spark is Airflow-on-Hadoop. Does anyone know what the difference is ?

Airflow doesn't have anything to do with data storage, movement or processing. It's a way to chain commands together in such a way so that you can define "do Z after Y a Z finish", for example. Many people use it like a nice version of cron with a UI, alerting, and retries.

so - celery + spark ? or just Celery Canvas ? (http://docs.celeryproject.org/en/latest/userguide/canvas.htm...)

P.S. I'm not trolling - I'm genuinely trying to get a sense of why and when would I use Airflow. Is it a point of scalability, of productivity , etc ?

For example - the positioning of spark is simple: scalability. Celery is also very clear: simplicity with good enough robustness if using the rabbitmq backend.

what does Airflow do differently ?

Re: Airflow and the Future of Data Engineering: A Q&A

#43

Post by colinwilson »

gregn610 wrote: post #36

How relevant are Airflow and similar to those of us who aren't operating at unicorn scale but are shuffling hundreds of CSVs & Excels and wrangling RDBMS with SQL?

I would say it can be used at it's simplest as a replacement for cron. It supports running programs on a schedule and you can set concurrency rules, SLAs, and triggers around even single commands or programs. You also get nice graphs of task run time and email alerts if jobs take longer than the SLA you've set.

Re: Airflow and the Future of Data Engineering: A Q&A

#44

Post by tedmiston »

Hey HN, I co-wrote the article with Maxime. A little late to the party here but happy to answer any questions you might have about it. I'll send him the link as well.

By the way, thank you to Maxime for sharing his thoughts, and the Astronomer team for contributing great questions.

Re: Airflow and the Future of Data Engineering: A Q&A

#45

Post by amalag »

gregn610 wrote: post #36

How relevant are Airflow and similar to those of us who aren't operating at unicorn scale but are shuffling hundreds of CSVs & Excels and wrangling RDBMS with SQL?

I have used Talend in the past, sounds like a fit. But this seems to fit a different need around job management.

Re: Airflow and the Future of Data Engineering: A Q&A

#46

Post by detroitcoder »

throwaway_374 wrote: post #26

Minor gripe - why can I not execute an entire DAG (end to end) from the UI? Also trying to execute single tasks from the UI using the "run" functionality gives a CeleryExecutor requirement error... sorry, I know this isn't the help forums but it sounds like the most trivial tasks were overlooked.

Actually you can but it is a bit clunky. Go to Browse > Dag Runs and select the 'Create' tab. This pulls up a form where you type in the Dag ID, enter the start time (set to now but keep in mind it is the local time of the web server), and a Run ID.

Totally agree there should be both a simple Start Run and Stop Run button.

Re: Airflow and the Future of Data Engineering: A Q&A

#47

Post by gregn610 »

amalag wrote: post #45
gregn610 wrote: post #36

How relevant are Airflow and similar to those of us who aren't operating at unicorn scale but are shuffling hundreds of CSVs & Excels and wrangling RDBMS with SQL?

I have used Talend in the past, sounds like a fit. But this seems to fit a different need around job management.

Talend makes my teeth grind. I don't understand why an ETL tool uses a strongly typed language for a foundation. The number of fun productive hours I've spent swapping chars to varchars, int to decimal & vice versa. In 2017 computers can read a registration plate from a blurry photograph and spot a criminal in a stadium, but a user puts apostrophe in a CSV file and schmoo leaks everywhere.

Re: Airflow and the Future of Data Engineering: A Q&A

#48

Post by caravel »

tedmiston wrote: post #44

Hey HN, I co-wrote the article with Maxime. A little late to the party here but happy to answer any questions you might have about it. I'll send him the link as well. By the way, thank you to Maxime for sharing his thoughts, and the Astronomer team for contributing great questions.

Maxime reporting for duty here, I'll go through the thread and answer questions / add comments.

Re: Airflow and the Future of Data Engineering: A Q&A

#49

Post by smooc »

sandGorgon wrote: post #42
gallamine wrote: post #40

Earlier quoted context omitted.

Airflow doesn't have anything to do with data storage, movement or processing. It's a way to chain commands together in such a way so that you can define "do Z after Y a Z finish", for example. Many people use it like a nice version of cron with a UI, alerting, and retries.

so - celery + spark ? or just Celery Canvas ? ( http://docs.celeryproject.org/en/latest/userguide/canvas.htm... ) P.S. I'm not trolling - I'm genuinely trying to get a sense of why and when would I use Airflow. Is it a point of scalability, of productivity , etc ? For example - the positioning of spark is simple: scalability. Celery is also very clear: simplicity with good enough robustness if using the rabbitmq back…

Airflow uses Celery to horizontally scale its execution. The Airflow scheduler takes care of what tasks to run in what order, but also what to do when they fail, need to retry, don't need to run at all, backfill the past etc.

Spark for Airflow is just one of the engines where a transformation of data can happen.

Re: Airflow and the Future of Data Engineering: A Q&A

#50

Post by tedmiston »

detroitcoder wrote: post #46
throwaway_374 wrote: post #26

Minor gripe - why can I not execute an entire DAG (end to end) from the UI? Also trying to execute single tasks from the UI using the "run" functionality gives a CeleryExecutor requirement error... sorry, I know this isn't the help forums but it sounds like the most trivial tasks were overlooked.

Actually you can but it is a bit clunky. Go to Browse > Dag Runs and select the 'Create' tab. This pulls up a form where you type in the Dag ID, enter the start time (set to now but keep in mind it is the local time of the web server), and a Run ID. Totally agree there should be both a simple Start Run and Stop Run button.

What I've been doing when debugging is flipping the DAG switch to off, clearing the first task in the DAG so the whole thing re-runs, then back to on.
Post reply on HN