I need to deal with system with complex dependency relationship. The system need to stop executing when some step fails. I used to look at Airflow. I remember at the time it was the first result if you Google DAG something. But it has the same problem as data-engineer centric system that it was somewhat over-engineered and it relies too much on a center server. It’s heavily relying on a Python runtime is also something I don’t like, feels like everything else is a second-class citizen. The nature of our job is that we have deal with complex legacy codes, some Fortran program, some need a conda environment etc. Later I found what I should look at is just some makefile-like system, and I settled on Snakemake, which has a nice DSL that forces you to be explicit about input / output / etc.. Probably Airflow is just not the right tool for a one man team.
Airflow's Problem
41–50 of 126 posts
Re: Airflow's Problem
#42Snowflake is the future. We shouldn't be writing code at all, we should be throwing data into a big hole in the ground and then querying the hole, or attaching a query to the hole so as data goes in the query does something wjth it, and chaining those queries. The fact that anyone is writing anything more complex than SQL to do this is a failure of imagination. Snowflake is intended to remove all the unnecessary engineering from you just putting data somewhere and doing something useful with it easily.
Re: Airflow's Problem
#43Is there any love for the Argo [1] project suite (Workflows, Events, CD) for this type of use case? I haven’t tried it out myself yet however it does look interesting.
Re: Airflow's Problem
#44To address a point the author makes: I’m entirely unconvinced the “shift left” mentality of data democracy (aka business operators should write sql) is actually shifting left or a worthy path to pursue for most businesses. More recently this 2010s fad seems to be dying and in favor we’re seeing centralized data efforts that produce data products. One of the most significant pitfalls of data is failing to interrogate…
With all credit due to Google's excellent and under-appreciated paper Machine Learning: The High Interest Credit Card of Technical Debt [1], I submit that Big Data is the high interest home equity line of credit of business operations debt.
It's not that big data tools aren't useful. It's that, when you just start amassing huge piles of data without a clear up-front plan for how it will be used, and assume that a whole bunch of people who have never heard of sampling bias or multiple comparisons bias or Coase's Law [2] can figure out what to do with it later, you're setting yourself up for a Bad Time.
1: https://research.google/pubs/pub43146/
2: "If you torture the data long enough, it will confess."Re: Airflow's Problem
#45Author here - appreciate the comments and reads. To add a bit of color -- I spent about a month looking into orchestrators to migrate Whatnot's data platform onto earlier this year, and it was a miserable experience. We were on AWS Managed Airflow, but to stay on it and have a solid platform, I would have been writing Github Actions for CI/CD, standing up ECR and IAM roles with Terraform, setting up EKS to run Kubern…
>We were on AWS Managed Airflow, but to stay on it and have a solid platform, I would have been writing Github Actions for CI/CD, standing up ECR and IAM roles with Terraform, setting up EKS to run Kubernetes jobs, managing infra monitoring with Datadog, etc., etc.
This sounds like an issue not with Airflow but with integration.
Re: Airflow's Problem
#46Dismissing Airflow for not being Astronomer is like dismissing Linux for not having the capabilities of a large-scale hypervisor. Replace “Airflow” with “Linux,” “data engineers” with “systems programmers,” and “Astronomer” with your hypervisor of choice (Xen/VMWare/etc.), and you can see how absurd the author’s point is: My problem is that ~Airflow~ Linux was not designed to address [high-level systems architecture]…
Managed Airflow doesn't even solve any of the author's outlined frustrations. It keeps the "obscene" syntax, it's still stateless, it's not "decentralized" etc.
Honestly, the article is so disingenuous that it comes off like a paid-for puff piece for Astronomer. It's the article-equivalent of the late-night infomercial guy who rips open a bag of potato chips like the hulk because he doesn't have this special tool that's just four easy payments of $9.99.
Re: Airflow's Problem
#47I despise airflow and how cemented it is as data infrastructure. It such a useful and basic concept but a nightmare to manage, and it works like junk. It's taken me 3 separate jobs over 7 years to realize that it's probably not our fault. Everyone seems to struggle with the same things: flaky scheduler that is slow to run tasks, confusing and redundant sounding settings that apply at up to three different levels (environment, job, task). It invites less experienced users to write a sea of spaghetti code in a monolithic DAGs repo. People wind up doing heavy data munging in python operators, which clobbers scalability and reliability. It also can't handle a large number of parallel tasks or frequent runs. It seems to have miserable scalability for the resources given, and bad controls for auto scaling. The UI feels dated and unintuitive. XComs seem useful to everyone but work like crap and actually an anti-pattern.
I've also tried it on Cloud Composer (google managed) and automated upgrades always trashed the cluster. It's not well designed for GKE because it writes logs to files and requires stateful sets. Testing the code is a huge burden due to the vast environment and dependencies needed to make it work locally.
I'm eager to rid my life of it and test out temporal for some of the high concurrency/frequency cases we have.
Re: Airflow's Problem
#48Author here - appreciate the comments and reads. To add a bit of color -- I spent about a month looking into orchestrators to migrate Whatnot's data platform onto earlier this year, and it was a miserable experience. We were on AWS Managed Airflow, but to stay on it and have a solid platform, I would have been writing Github Actions for CI/CD, standing up ECR and IAM roles with Terraform, setting up EKS to run Kubern…
In your investigation did you try out Flyte at all?
Re: Airflow's Problem
#49Author here - appreciate the comments and reads. To add a bit of color -- I spent about a month looking into orchestrators to migrate Whatnot's data platform onto earlier this year, and it was a miserable experience. We were on AWS Managed Airflow, but to stay on it and have a solid platform, I would have been writing Github Actions for CI/CD, standing up ECR and IAM roles with Terraform, setting up EKS to run Kubern…
>We were on AWS Managed Airflow, but to stay on it and have a solid platform, I would have been writing Github Actions for CI/CD, standing up ECR and IAM roles with Terraform, setting up EKS to run Kubernetes jobs, managing infra monitoring with Datadog, etc., etc. This sounds like an issue not with Airflow but with integration.
Yep, that's what I tried to point out in the article.
Re: Airflow's Problem
#50Author here - appreciate the comments and reads. To add a bit of color -- I spent about a month looking into orchestrators to migrate Whatnot's data platform onto earlier this year, and it was a miserable experience. We were on AWS Managed Airflow, but to stay on it and have a solid platform, I would have been writing Github Actions for CI/CD, standing up ECR and IAM roles with Terraform, setting up EKS to run Kubern…
In your investigation did you try out Flyte at all?
Nope -- just MWAA, Astronomer, Dagster Cloud, and Prefect Cloud. In the past I used Argo Workflows pretty extensively and have talked about its pros and cons here: https://www.youtube.com/watch?v=-cyr_kL-9fc