Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

The Unbundling of Airflow

blog.fal.ai

Post reply on HN
Print view View original

31–40 of 60 posts

Re: The Unbundling of Airflow

#31

Post by the_af »

samdjstephens wrote: post #3
fdgsdfogijq wrote: post #2

This post is hard to follow. But I'll give my unsolicited opinion on airflow: Its too complex to run as a single team and there are far better tools out there for scheduling. Airflow only makes sense when you need complex logic surrounding when to run jobs, how to backfill, when to backfill, and complex dependency trees. Otherwise, you are much better off with something like AWS step functions.

Interesting, I wouldn’t say that I’ve found it difficult to run in even a small team. The problem I’ve always had with Airflow has been with non-cron-like use cases, for example data pipelines kicked off when some event occurs. Sensors were often an awkward fit and the HTTP API was quite immature back when I was using it

Agreed about sensors. We still have some trouble figuring them out and understanding why they sometimes don't trigger when they should.

Re: The Unbundling of Airflow

#32

Post by liveoneggs »

fmakunbound wrote: post #9

It's ok, but seems to be a bit too complex for what it does. It was pretty janky running it locally (pegged the CPU), and now that we have it in MWAA we've got several support issues on it with AWS for unkillable task instances and scheduler problems.

I also suffered that cpu bug many years ago but I'd hope it has been fixed by now! The scheduler stealing all of the cycles.....

Re: The Unbundling of Airflow

#33

Post by liveoneggs »

ajoseps wrote: post #4
fdgsdfogijq wrote: post #2

This post is hard to follow. But I'll give my unsolicited opinion on airflow: Its too complex to run as a single team and there are far better tools out there for scheduling. Airflow only makes sense when you need complex logic surrounding when to run jobs, how to backfill, when to backfill, and complex dependency trees. Otherwise, you are much better off with something like AWS step functions.

do you have recommendations on alternatives that are not tied to a cloud provider?

jenkins

Re: The Unbundling of Airflow

#34

Post by the_af »

throwaway984393 wrote: post #24

You can "unbundle" Airflow into different components. What is it called when you take one thing and break it into many pieces? Distributed ( sometimes decentralized) computing. What do you get when you take a single system and distribute/decentralize it? Complexity. And what's the best way to simplify complexity? Consolidate the complexity into one system. The Circle of Computing Complexity.

The author even mentions they hope to see this consolidated into dbt Cloud right there at the end of the article!

Re: The Unbundling of Airflow

#35

Post by speed_spread »

This is fine and will allow Airflow to focus on it's core functionality of being a distributed job scheduler.

FWIW, last I looked at Airflow I thought the schedule+task model could be made tighter as their was numerous ways to enter inconsistent states. For example, changing the schedule after tasks had already been run would allow to rerun jobs (in the past) at dates that were never scheduled in the first place.

Re: The Unbundling of Airflow

#36

Post by the_af »

As a newcomer to the world of data, I have no strong opinions about Airflow. It replaced a bunch of disparate cron jobs, so it's definitely better than what was there before.

There are things I like and things I don't about it. The UI is awful -- I don't know anyone that likes it, unlike what the article states. I like that it's centralized and that it's all Python code.

Deploying it and fine-tuning the config for a variety of workloads can be a pain. Sometimes sensors don't work right. Tasks sometimes get evicted and killed for obscure reasons. Zombie tasks are a pain big enough you'll see plenty of requests for help online.

That said, replacing it with a bunch of disparate tools again? Seems like a step backwards. And now instead of a single tool, your org has to vet, secure, understand and monitor a bunch of different tools? It's bad enough with only one...

What am I missing?

PS: data analysis/engineering as a field seems new and immature enough that, in my humble opinion, we should be focusing on developing good practices and theory, instead of deprecating existing (and pretty recent) tech at an ever increasing pace.

Re: The Unbundling of Airflow

#37

Post by shmoogy »

fdgsdfogijq wrote: post #2

This post is hard to follow. But I'll give my unsolicited opinion on airflow: Its too complex to run as a single team and there are far better tools out there for scheduling. Airflow only makes sense when you need complex logic surrounding when to run jobs, how to backfill, when to backfill, and complex dependency trees. Otherwise, you are much better off with something like AWS step functions.

I manage and run our airflow instance - outside of migrating from 1.X to 2.x I haven't really had any problems. Learning curve was a bit higher than I hoped, but being able to set tasks downstream and backfill is so much nicer than regular cron / windows task manager script running.

Re: The Unbundling of Airflow

#38

Post by rhacker »

the_af wrote: post #36

As a newcomer to the world of data, I have no strong opinions about Airflow. It replaced a bunch of disparate cron jobs, so it's definitely better than what was there before. There are things I like and things I don't about it. The UI is awful -- I don't know anyone that likes it, unlike what the article states. I like that it's centralized and that it's all Python code. Deploying it and fine-tuning the config for a…

Thanks for saying this. I also have been tasked to introduce airflow at my company. I decided to use 2.0 so it's more python dags. But for the most part the dags are JUST triggered via web service by other processes.

so... it's nothing more than processing plus a queue. I mean we already have rabbit and typescript. We also already have Typescript + Agenda (over mongo).

We have gotten to the point where a single company is implementing queuing at least 4 different ways because "microservices".

Re: The Unbundling of Airflow

#39

Post by hulahoof »

zukzuk wrote: post #10
fdgsdfogijq wrote: post #2

This post is hard to follow. But I'll give my unsolicited opinion on airflow: Its too complex to run as a single team and there are far better tools out there for scheduling. Airflow only makes sense when you need complex logic surrounding when to run jobs, how to backfill, when to backfill, and complex dependency trees. Otherwise, you are much better off with something like AWS step functions.

Everyone's context is different, but I've found the exact opposite to be true. Airflow is simple and dumb enough that it can be easily understood and managed by a small team, but it's also flexible and powerful enough that we can't come up with a good enough reason to switch to anything else.* *We are, however, becoming more and more reliant on dbt, and the article makes a good point about Airflow providing no visibi…

This reflects how I often deploy Airflow as well (usually on GCP as Composer)

We use DBT to manage the DAG for the BQ transformations, put this in a container and deploy it into the kubernetes cluster that airflow is running on as a single node.

Airflow can then handle the scheduling and DAG nodes for non DWH dependencies such as loading/checking for files, kicking off tasks that need to run after the DWH refresh and the like.

I find once it is set up it is extremely easy for small teams to follow the pattern, and the single view of all the pipelines running is a great benefit - as well as handling the logic around last successful runs etc., that would need to be implemented manually if using simple cron jobs.

Re: The Unbundling of Airflow

#40

Post by hulahoof »

jimmytucson wrote: post #27

If I put each select statement in its own Airflow task, I get the same lineage dbt gives me, except I can see it and administer to it alongside all my other E and L-type tasks. Also, I can write my T in plain ol’ SQL (granted, with some jinja) instead of this dbt-QL that I can’t copy and paste into my database console or share with a non-dbt user. So, folks who have adopted dbt: what am I missing by being a fuddy-dud…

I don't think you are missing anything, but allowing DBT to contain all the models that make up your various pipelines and reference each other mean that you can schedule your various pipelines at different cadences and use tags to refresh the relevant DBT models from a single code base.

It sounds like in your approach this would be writing this dependency logic into each DAG you schedule on airflow.

In the same way you would interpolate your jinja SQL before copying it into the database, you would use dbt compile or the output from a dbt run from the target/ folder and copy that SQL into your DB console or to share.

EDIT: This means your T is a single airflow node in each DAG, though I then still use airflow for the E/L tasks around it

Post reply on HN