Airflow and the Future of Data Engineering: A Q&A • Hacker News Board

Re: Airflow and the Future of Data Engineering: A Q&A

#61

Post by caravel » Wed, Mar 01, 2017, 5:01 PM UTC

batmansmk wrote: post #15

A lot of name dropping and unprovable statements. I'm really interested by the domain and progress, but Airflow needs to be more generous in real information and less in marketing bs. Can someone share a more introductory article about what makes Airflow different from the current state of the art?

[author] sorry you feel that way. I understand that the section on other similar tools is controversial, especially if you work on one of those. I'm repeating what I hear being very active in this community, and answering the question that was asked to the best of my knowledge. I'm open to editing the article with better information if anyone wants to share more insight.

As mentioned about Luigi, I do not have the whole story, one fact I know is that someone from Spotify gave a talk I did not attend in NYC at an Airflow meetup, and I've heard the original author had left the company. Those are provable statements, happy to debunk on the article if needed.

What do you mean by [current state of the Art]?

Re: Airflow and the Future of Data Engineering: A Q&A

#62

Post by caravel » Wed, Mar 01, 2017, 5:11 PM UTC

sandGorgon wrote: post #42

gallamine wrote: post #40

Earlier quoted context omitted.

Airflow doesn't have anything to do with data storage, movement or processing. It's a way to chain commands together in such a way so that you can define "do Z after Y a Z finish", for example. Many people use it like a nice version of cron with a UI, alerting, and retries.

so - celery + spark ? or just Celery Canvas ? ( http://docs.celeryproject.org/en/latest/userguide/canvas.htm... ) P.S. I'm not trolling - I'm genuinely trying to get a sense of why and when would I use Airflow. Is it a point of scalability, of productivity , etc ? For example - the positioning of spark is simple: scalability. Celery is also very clear: simplicity with good enough robustness if using the rabbitmq back…

In a modern data team, Spark is just one of the type of job you may want to orchestrate. Typically as your company gets more tangled in data processing, you'll have many storage and compute engines that you'll have to orchestrate. Hive, MySQL, Presto, HBASE, map/reduce, Cascading/Scalding, scripts, external integrations, R, Druid, Redshift, miroservices, ...

Airflow allows you to orchestrate all of this and keep most of code and high level operation in one place.

Of course Spark has its own internal DAG and can somewhat act as Airflow and trigger some of these other things, but typically that breaks down as you have a growing array of Spark jobs and want to keep a holistic view.

Re: Airflow and the Future of Data Engineering: A Q&A

#63

Post by vira » Wed, Mar 01, 2017, 5:19 PM UTC

gregn610 wrote: post #47

amalag wrote: post #45

Earlier quoted context omitted.

I have used Talend in the past, sounds like a fit. But this seems to fit a different need around job management.

Talend makes my teeth grind. I don't understand why an ETL tool uses a strongly typed language for a foundation. The number of fun productive hours I've spent swapping chars to varchars, int to decimal & vice versa. In 2017 computers can read a registration plate from a blurry photograph and spot a criminal in a stadium, but a user puts apostrophe in a CSV file and schmoo leaks everywhere.

Strong typing helps. Keep in mind enterprise ETL tools are designed to move data from oltp to olap databases, which are often strongly typed as well.

Re: Airflow and the Future of Data Engineering: A Q&A

#64

Post by [deleted] » Wed, Mar 01, 2017, 5:19 PM UTC

gregn610 wrote: post #47

amalag wrote: post #45

Earlier quoted context omitted.

I have used Talend in the past, sounds like a fit. But this seems to fit a different need around job management.

Talend makes my teeth grind. I don't understand why an ETL tool uses a strongly typed language for a foundation. The number of fun productive hours I've spent swapping chars to varchars, int to decimal & vice versa. In 2017 computers can read a registration plate from a blurry photograph and spot a criminal in a stadium, but a user puts apostrophe in a CSV file and schmoo leaks everywhere.

[deleted]

Re: Airflow and the Future of Data Engineering: A Q&A

#65

Post by LevonK » Wed, Mar 01, 2017, 5:21 PM UTC

Disney Studios uses Azkaban because it's language agnostic, and it believes that in the data space there is a huge advantage to static (and strong, but that's beside the point in this argument) typing in the data space.

The language agnostic aspect means that non software engineers can also use the orchestration platform for runbook automation.

Re: Airflow and the Future of Data Engineering: A Q&A

#66

Post by sandGorgon » Wed, Mar 01, 2017, 5:24 PM UTC

caravel wrote: post #62

sandGorgon wrote: post #42

Earlier quoted context omitted.

so - celery + spark ? or just Celery Canvas ? ( http://docs.celeryproject.org/en/latest/userguide/canvas.htm... ) P.S. I'm not trolling - I'm genuinely trying to get a sense of why and when would I use Airflow. Is it a point of scalability, of productivity , etc ? For example - the positioning of spark is simple: scalability. Celery is also very clear: simplicity with good enough robustness if using the rabbitmq back…

In a modern data team, Spark is just one of the type of job you may want to orchestrate. Typically as your company gets more tangled in data processing, you'll have many storage and compute engines that you'll have to orchestrate. Hive, MySQL, Presto, HBASE, map/reduce, Cascading/Scalding, scripts, external integrations, R, Druid, Redshift, miroservices, ... Airflow allows you to orchestrate all of this and keep most…

that is an incredibly lucid answer. That should be the first line on the Airflow project.

Re: Airflow and the Future of Data Engineering: A Q&A

#67

Post by caravel » Wed, Mar 01, 2017, 5:33 PM UTC

botswana99 wrote: post #57

[Bias Alert: I'm Head Chef of DataKitchen]. Our perspective is that the DAG abstraction should not apply only to data engineering, but the whole analytic process of data engineering, data science, and data visualization. Analytic teams love to work with their favorite tools -- Python, SQL, ETL, Jupyter, R, Tableau, Alteryx, etc. The question is how do you get those diverse teams and tools to work together to deliver…

[Bias Alert: author of Airflow] can confirm that Airflow allows you to incorporate all of the seven steps, and more as an open platform.

At Airbnb Airflow is far from being limited to data engineering. All the batch scheduling goes through Airflow and many team (data science, analysts, data infra, ML infra, engineering as a whole, ...) uses Airflow in all sorts of ways.

Airflow has a solid story in terms of reusable components, from extendable abstractions (operator, hooks, executors, macros, ...) all the way to computation frameworks.

Re: Airflow and the Future of Data Engineering: A Q&A

#68

Post by jaz46 » Wed, Mar 01, 2017, 5:34 PM UTC

One of the important paradigms that I think Luigi and Airflow miss is that they treat pipelines as a DAG of tasks, when it really should be thought of as a DAG of Data.

It's a subtle difference, but has huge impacts when you're trying to dynamically scale tasks based on cluster resources and track data lineage throughout your system.(Disclosure: I'm the founder of Pachyderm[0], a containerized data pipeline framework where we version control data in this way).

Check out Samuel Lampa's post[1] about dynamically scaling data pipelines for more details.

[0] github.com/pachyderm/pachyderm [1] http://bionics.it/posts/dynamic-workflow-scheduling

Re: Airflow and the Future of Data Engineering: A Q&A

#69

Post by batbomb » Wed, Mar 01, 2017, 5:41 PM UTC

dataops wrote: post #32

Data engineering is converging under the umbrella of DataOps. For those interested, there's a DataOps Summit in Boston this June https://www.dataopssummit.com/

This is just Data Management, a term which predates "DataOps" by more than a decade in both research and enterprise. I don't really think it needs a rebranding.

Re: Airflow and the Future of Data Engineering: A Q&A

#70

Post by gdulli » Wed, Mar 01, 2017, 5:43 PM UTC

tedmiston wrote: post #60

gdulli wrote: post #41

Earlier quoted context omitted.

I've started working on a code generator for Airflow. Not primarily because I needed dynamic jobs, but more because I didn't want to keep writing the Airflow boilerplate. I imagine I'll eventually need to add some sort of management system to move these dynamic jobs in and out of Airflow to keep them from bloating the database or cluttering the UI.

I'd be interested in hearing more about your project and if you plan to open source. Are you using something like Cookiecutter or Yeoman, or a different level of abstraction?

My effort is a collection of custom operators for operations that we use very commonly (run a query, python script, or mapreduce job) and a code generator that takes a simple text spec describing a set of operations and generates the Python DAG script. I'm not familiar with Cookiecutter or Yeoman but generating Python code myself hasn't been complicated. Open-sourcing it isn't an option right now with my current employer.