41–50 of 60 posts
Earlier quoted context omitted.
Exactly my thoughts as well. I have one point where I can see if all the remote services that I am using are operating correctly. I don't need to connect to various other apps to figure this out.
This is why we moved to airflow vs lots of Cron jobs. Centralized place to look, logging, etc.
I found Airflow to be extremely buggy when I started deploying it for use with a small/medium sized team. Did anybody else have a similar impression, or has it gotten better with later versions?
If I put each select statement in its own Airflow task, I get the same lineage dbt gives me, except I can see it and administer to it alongside all my other E and L-type tasks. Also, I can write my T in plain ol’ SQL (granted, with some jinja) instead of this dbt-QL that I can’t copy and paste into my database console or share with a non-dbt user. So, folks who have adopted dbt: what am I missing by being a fuddy-dud…
Well written. I think that airflow is being enforced in organizations as the main orchestrator even though it's not always the right too for the job. In addition, organizations has to enforce a micro-services approach to have modular components. Besides that managing those frameworks is a nightmare. We built Ploomber ( https://github.com/ploomber/ploomber ) specifically for this reason, modular components and easy de…
I think you're 100% right that the tasks that can be accomplished in Airflow are currently being unbundled by tools in the modern data stack, but that doesn't erase the need for tools like Airflow. Sure, you can now write less code to load your data, transform it, and send it out to other tools. As the unbundling occurs, the end result is more fragmentation and fragility in how teams manage their data. Data teams I t…
Isn't the main selling point of airflow the bundling in the first place? Why would you want many different specialized tools to manage scheduled tasks?
Thanks.
This post is hard to follow. But I'll give my unsolicited opinion on airflow: Its too complex to run as a single team and there are far better tools out there for scheduling. Airflow only makes sense when you need complex logic surrounding when to run jobs, how to backfill, when to backfill, and complex dependency trees. Otherwise, you are much better off with something like AWS step functions.
Really? Which ones? The only thing vaguely fitting this case is Jenkins, but using Jenkins to run ETL/ELT is a serious impedance mismatch.
As a newcomer to the world of data, I have no strong opinions about Airflow. It replaced a bunch of disparate cron jobs, so it's definitely better than what was there before. There are things I like and things I don't about it. The UI is awful -- I don't know anyone that likes it, unlike what the article states. I like that it's centralized and that it's all Python code. Deploying it and fine-tuning the config for a…
Airflow is... not amazing. But by the standards of horrible enterprise software we've all been subjected to, it's not that bad.
If you're complaining about Airflow, wait for the day you're forced to use an internally built database client.
That's Afghanistan.
Our proprietary AWS wrapper takes 45 damn minutes on a good day to allocate a VM. The AMI is built in two minutes. TWO.
I'm sure in 5 years Dagster and Prefect will have improved gradually in lots of incremental ways. For now Airflow is pretty solid.
This post is hard to follow. But I'll give my unsolicited opinion on airflow: Its too complex to run as a single team and there are far better tools out there for scheduling. Airflow only makes sense when you need complex logic surrounding when to run jobs, how to backfill, when to backfill, and complex dependency trees. Otherwise, you are much better off with something like AWS step functions.
> there are far better tools out there for scheduling Really? Which ones? The only thing vaguely fitting this case is Jenkins, but using Jenkins to run ETL/ELT is a serious impedance mismatch.
But yes, I'm confused. Triggering a dag and having it exit based on complex logic is a perfectly normal pattern.
As a newcomer to the world of data, I have no strong opinions about Airflow. It replaced a bunch of disparate cron jobs, so it's definitely better than what was there before. There are things I like and things I don't about it. The UI is awful -- I don't know anyone that likes it, unlike what the article states. I like that it's centralized and that it's all Python code. Deploying it and fine-tuning the config for a…
What you're missing is that for much of enterprise software before Airflow, everything was steaming rubbish . Airflow is... not amazing. But by the standards of horrible enterprise software we've all been subjected to, it's not that bad . If you're complaining about Airflow, wait for the day you're forced to use an internally built database client. That's Afghanistan. Our proprietary AWS wrapper takes 45 damn minutes…