The Unbundling of Airflow
blog.fal.ai
The Unbundling of Airflow
1–10 of 60 posts
Re: The Unbundling of Airflow
#2This post is hard to follow. But I'll give my unsolicited opinion on airflow:
Its too complex to run as a single team and there are far better tools out there for scheduling. Airflow only makes sense when you need complex logic surrounding when to run jobs, how to backfill, when to backfill, and complex dependency trees. Otherwise, you are much better off with something like AWS step functions.
Re: The Unbundling of Airflow
#3This post is hard to follow. But I'll give my unsolicited opinion on airflow: Its too complex to run as a single team and there are far better tools out there for scheduling. Airflow only makes sense when you need complex logic surrounding when to run jobs, how to backfill, when to backfill, and complex dependency trees. Otherwise, you are much better off with something like AWS step functions.
Interesting, I wouldn’t say that I’ve found it difficult to run in even a small team.
The problem I’ve always had with Airflow has been with non-cron-like use cases, for example data pipelines kicked off when some event occurs. Sensors were often an awkward fit and the HTTP API was quite immature back when I was using it
Re: The Unbundling of Airflow
#4This post is hard to follow. But I'll give my unsolicited opinion on airflow: Its too complex to run as a single team and there are far better tools out there for scheduling. Airflow only makes sense when you need complex logic surrounding when to run jobs, how to backfill, when to backfill, and complex dependency trees. Otherwise, you are much better off with something like AWS step functions.
do you have recommendations on alternatives that are not tied to a cloud provider?
Re: The Unbundling of Airflow
#5This post is hard to follow. But I'll give my unsolicited opinion on airflow: Its too complex to run as a single team and there are far better tools out there for scheduling. Airflow only makes sense when you need complex logic surrounding when to run jobs, how to backfill, when to backfill, and complex dependency trees. Otherwise, you are much better off with something like AWS step functions.
do you have recommendations on alternatives that are not tied to a cloud provider?
We are trying to build something like this at https://www.magniv.app/.
Would love to have you join our beta if you are interested!
Re: The Unbundling of Airflow
#6High Resolution Version of the diagram if anyone is interested
https://drive.google.com/file/d/1btZ0yck9SdgsUdNom0WXgHcSQvO...
Re: The Unbundling of Airflow
#7I love Airflow. Plenty of data businesses I've built are nothing more than just one DAG.
As for the article, I don't think we are yet at the point in which a competing stack comprised of individual specialized components do things better since Airflow is more than the sum of its parts imho.
Re: The Unbundling of Airflow
#8This post is hard to follow. But I'll give my unsolicited opinion on airflow: Its too complex to run as a single team and there are far better tools out there for scheduling. Airflow only makes sense when you need complex logic surrounding when to run jobs, how to backfill, when to backfill, and complex dependency trees. Otherwise, you are much better off with something like AWS step functions.
I run Airflow even for my local trading setup. For large teams, I often go with managed solutions like Astronomer.
Re: The Unbundling of Airflow
#9It's ok, but seems to be a bit too complex for what it does. It was pretty janky running it locally (pegged the CPU), and now that we have it in MWAA we've got several support issues on it with AWS for unkillable task instances and scheduler problems.
Re: The Unbundling of Airflow
#10This post is hard to follow. But I'll give my unsolicited opinion on airflow: Its too complex to run as a single team and there are far better tools out there for scheduling. Airflow only makes sense when you need complex logic surrounding when to run jobs, how to backfill, when to backfill, and complex dependency trees. Otherwise, you are much better off with something like AWS step functions.
Everyone's context is different, but I've found the exact opposite to be true. Airflow is simple and dumb enough that it can be easily understood and managed by a small team, but it's also flexible and powerful enough that we can't come up with a good enough reason to switch to anything else.*
*We are, however, becoming more and more reliant on dbt, and the article makes a good point about Airflow providing no visibility for what's going on in a dbt node. So we're ending up with an increasingly simpler Airflow dag, with most of the complexity hidden inside a single dbt node.