I found Airflow to be extremely buggy when I started deploying it for use with a small/medium sized team. Did anybody else have a similar impression, or has it gotten better with later versions?
The Unbundling of Airflow
21–30 of 60 posts
Re: The Unbundling of Airflow
#22This post is hard to follow. But I'll give my unsolicited opinion on airflow: Its too complex to run as a single team and there are far better tools out there for scheduling. Airflow only makes sense when you need complex logic surrounding when to run jobs, how to backfill, when to backfill, and complex dependency trees. Otherwise, you are much better off with something like AWS step functions.
Everyone's context is different, but I've found the exact opposite to be true. Airflow is simple and dumb enough that it can be easily understood and managed by a small team, but it's also flexible and powerful enough that we can't come up with a good enough reason to switch to anything else.* *We are, however, becoming more and more reliant on dbt, and the article makes a good point about Airflow providing no visibi…
I'm not too familiar with the use of dbt but what was the reason you chose to have a single dbt node rather than translating the dependencies into an airflow dag?
Re: The Unbundling of Airflow
#23[deleted]
Re: The Unbundling of Airflow
#24You can "unbundle" Airflow into different components. What is it called when you take one thing and break it into many pieces? Distributed (sometimes decentralized) computing. What do you get when you take a single system and distribute/decentralize it? Complexity. And what's the best way to simplify complexity? Consolidate the complexity into one system.
The Circle of Computing Complexity.
Re: The Unbundling of Airflow
#25Well written. I think that airflow is being enforced in organizations as the main orchestrator even though it's not always the right too for the job. In addition, organizations has to enforce a micro-services approach to have modular components. Besides that managing those frameworks is a nightmare. We built Ploomber (https://github.com/ploomber/ploomber) specifically for this reason, modular components and easy deployments. It standardize your pipelines and allows you to deploy seamlessly on Airflow, Argo (Kubernetes), Kubeflow and cloud providers.
Re: The Unbundling of Airflow
#26I think you're 100% right that the tasks that can be accomplished in Airflow are currently being unbundled by tools in the modern data stack, but that doesn't erase the need for tools like Airflow. Sure, you can now write less code to load your data, transform it, and send it out to other tools. As the unbundling occurs, the end result is more fragmentation and fragility in how teams manage their data. Data teams I t…
Isn't the main selling point of airflow the bundling in the first place? Why would you want many different specialized tools to manage scheduled tasks?
Exactly my thoughts as well.
I have one point where I can see if all the remote services that I am using are operating correctly. I don't need to connect to various other apps to figure this out.
Re: The Unbundling of Airflow
#27If I put each select statement in its own Airflow task, I get the same lineage dbt gives me, except I can see it and administer to it alongside all my other E and L-type tasks.
Also, I can write my T in plain ol’ SQL (granted, with some jinja) instead of this dbt-QL that I can’t copy and paste into my database console or share with a non-dbt user.
So, folks who have adopted dbt: what am I missing by being a fuddy-duddy?
Re: The Unbundling of Airflow
#28Since this might catch the eye of someone knowledgeable … Does anyone know if the community docker images for airflow can be run using podman?
Do you mean like the Docker operator? Or the actual Airflow images themselves?
On the first one, I suspect "probably", on the second "yes".
Re: The Unbundling of Airflow
#29I have used airflow with two different organizations over the past couple years. When we had a complex orchestration with critical pipelines and enough human-power to manage the system, it was great. Trying to deploy it for a small team with no critical pipelines has been overkill and we recently migrated to dagster which is still in beta but accomplishes 90% of what airflow does with a much smaller footprint.
Re: The Unbundling of Airflow
#30Earlier quoted context omitted.
Isn't the main selling point of airflow the bundling in the first place? Why would you want many different specialized tools to manage scheduled tasks?
Exactly my thoughts as well. I have one point where I can see if all the remote services that I am using are operating correctly. I don't need to connect to various other apps to figure this out.
This is why we moved to airflow vs lots of Cron jobs. Centralized place to look, logging, etc.