Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Airflow's Problem

stkbailey.substack.com

Post reply on HN
Print view View original

51–60 of 126 posts

Re: Airflow's Problem

#51

Post by htrp »

mumblemumble wrote: post #44
llambda wrote: post #9

To address a point the author makes: I’m entirely unconvinced the “shift left” mentality of data democracy (aka business operators should write sql) is actually shifting left or a worthy path to pursue for most businesses. More recently this 2010s fad seems to be dying and in favor we’re seeing centralized data efforts that produce data products. One of the most significant pitfalls of data is failing to interrogate…

With all credit due to Google's excellent and under-appreciated paper Machine Learning: The High Interest Credit Card of Technical Debt [1], I submit that Big Data is the high interest home equity line of credit of business operations debt. It's not that big data tools aren't useful. It's that, when you just start amassing huge piles of data without a clear up-front plan for how it will be used, and assume that a who…

> I submit that Big Data is the high interest home equity line of credit of business operations debt.

I like this but it's kinda like the payday loan of business operations.

Re: Airflow's Problem

#52

Post by TTPrograms »

mywittyname wrote: post #46
MontyCarloHall wrote: post #7

Dismissing Airflow for not being Astronomer is like dismissing Linux for not having the capabilities of a large-scale hypervisor. Replace “Airflow” with “Linux,” “data engineers” with “systems programmers,” and “Astronomer” with your hypervisor of choice (Xen/VMWare/etc.), and you can see how absurd the author’s point is: My problem is that ~Airflow~ Linux was not designed to address [high-level systems architecture]…

Managed Airflow doesn't even solve any of the author's outlined frustrations. It keeps the "obscene" syntax, it's still stateless, it's not "decentralized" etc. Honestly, the article is so disingenuous that it comes off like a paid-for puff piece for Astronomer. It's the article-equivalent of the late-night infomercial guy who rips open a bag of potato chips like the hulk because he doesn't have this special tool tha…

The new TaskFlow API has been part of AirFlow 2.0 since its release in 2020: https://airflow.apache.org/docs/apache-airflow/stable/tutori...

Re: Airflow's Problem

#53

Post by lopatin »

dusted wrote: post #27
lopatin wrote: post #21

Earlier quoted context omitted.

I'd suggest to raise the issue in the devlist or on GitHub Issues to get more visibility.

I'd be okay with the article headline on HN being "Apache Airflow's Problem" so that I know it's about a piece of apache software and not something interesting about airflow.

I think context matters and the title "Airflow's Problem" doesn't make much sense when talking about the physical phenomenon of flowing air.

Re: Airflow's Problem

#54

Post by stkbailey »

mywittyname wrote: post #46
MontyCarloHall wrote: post #7

Dismissing Airflow for not being Astronomer is like dismissing Linux for not having the capabilities of a large-scale hypervisor. Replace “Airflow” with “Linux,” “data engineers” with “systems programmers,” and “Astronomer” with your hypervisor of choice (Xen/VMWare/etc.), and you can see how absurd the author’s point is: My problem is that ~Airflow~ Linux was not designed to address [high-level systems architecture]…

Managed Airflow doesn't even solve any of the author's outlined frustrations. It keeps the "obscene" syntax, it's still stateless, it's not "decentralized" etc. Honestly, the article is so disingenuous that it comes off like a paid-for puff piece for Astronomer. It's the article-equivalent of the late-night infomercial guy who rips open a bag of potato chips like the hulk because he doesn't have this special tool tha…

Interesting, Astronomer was actually my last choice for orchestrator. We went with Dagster, but I didn't want to make the takeaway "Dagster solves these problems", because it doesn't directly. Astronomer was just the best foil for the "meta-orchestrator" space that seems to be evolving, and which _can_ address these problems.

Re: Airflow's Problem

#55

Post by theptip »

stkbailey wrote: post #40

Author here - appreciate the comments and reads. To add a bit of color -- I spent about a month looking into orchestrators to migrate Whatnot's data platform onto earlier this year, and it was a miserable experience. We were on AWS Managed Airflow, but to stay on it and have a solid platform, I would have been writing Github Actions for CI/CD, standing up ECR and IAM roles with Terraform, setting up EKS to run Kubern…

Thanks for the experience report - I have Dagster and Prefect on my shortlist to evaluate next time I need to build this, and Dagster seems the most promising, so it’s good to get another datapoint.

One Q - it seems to me that another possible solve (and probably how the big guys tend to do it) is to use a dataflow engine like Spark/Flink. Did you compare a managed platform like Google Dataproc? They also have serverless if you don’t want a heavy managed cluster, which might make this approach more viable for non-huge companies that wouldn’t utilize a min-spec cluster. (When I last evaluated this they didn’t have serverless which was a dealbreaker for my small scale).

Re: Airflow's Problem

#56

Post by byteflip »

marcinzm wrote: post #24
edumucelli wrote: post #23

More recent tools such as Dagster and Prefect have much more to offer. One simple example is communication between tasks. Airflow has a clunky system for that called XCom. The actual author of XCom says you probably should not use it due to the level of hackery it has under the hood [1]: On Dagster and Prefect you communicate between tasks as if you were writing pure Python. On Airflow on the other hand ... [1] https…

I'm not sure if linking to a talk from 2018 in 2022 for a project that is being actively worked on (a bunch of Python abstractions for xcom were added in 2.0) is fair.

Yes, curious if the Taskflow API introduced as part of Airflow 2.0 reduces this pain. It appears much easier/saner than working with XCOMs directly - less coupling and removes the need for lots of boilerplate code.

Re: Airflow's Problem

#57

Post by datavirtue »

itsoktocry wrote: post #34
llambda wrote: post #9

To address a point the author makes: I’m entirely unconvinced the “shift left” mentality of data democracy (aka business operators should write sql) is actually shifting left or a worthy path to pursue for most businesses. More recently this 2010s fad seems to be dying and in favor we’re seeing centralized data efforts that produce data products. One of the most significant pitfalls of data is failing to interrogate…

> that if you give everyone access all the time the magic will happen There's much ongoing discussion about this is the data world, often revolving around "self-service analytics". Unless you're talking about "our analysts don't have to clean data all the time", which, for a large enough organization makes sense, "self-service" for non-technical folks is futile and pointless. They need specific answers to specific qu…

They idea was that they were going to hire an army of data scientists and become google...magically.

Reality smacked that shit down hard. I left data engineering because the projects were all over the place, wildly undisciplined and unfocused.

You were lucky to have source control let alone an understanding from the business that these projects were in fact software development.

I switched back to software engineering because at least there is a faint realization that we are...building software.

I might go back when the dust clears.

"Why do we need to hire programmers...I thought we needed data engineers?"

"Because the data pipelines are all built with thousands of lines of code. Java, python, Fortran, you name it...and your job post only mentioned SQL and data modelling"

I could go on forever.

Re: Airflow's Problem

#58

Post by karog »

edumucelli wrote: post #23

More recent tools such as Dagster and Prefect have much more to offer. One simple example is communication between tasks. Airflow has a clunky system for that called XCom. The actual author of XCom says you probably should not use it due to the level of hackery it has under the hood [1]: On Dagster and Prefect you communicate between tasks as if you were writing pure Python. On Airflow on the other hand ... [1] https…

There's also Flyte, which is natively data aware and schedules tasks around data dependencies. The syntax is essentially pure python too.

Re: Airflow's Problem

#59

Post by llbeansandrice »

I'm kinda meh on this article, but it did lead me to this goldmine[1]. We don't use XCOMs at all so a lot of these aren't applicable but other parts absolutely are. We run Airflow at a pretty massive scale and not all of these boundaries were enforced so now it's a huge mess.

[1] https://towardsdatascience.com/apache-airflow-in-2022-10-rul...

Re: Airflow's Problem

#60

Post by awinder »

stkbailey wrote: post #40

Author here - appreciate the comments and reads. To add a bit of color -- I spent about a month looking into orchestrators to migrate Whatnot's data platform onto earlier this year, and it was a miserable experience. We were on AWS Managed Airflow, but to stay on it and have a solid platform, I would have been writing Github Actions for CI/CD, standing up ECR and IAM roles with Terraform, setting up EKS to run Kubern…

“We were on AWS Managed Airflow, but to stay on it and have a solid platform, I would have been writing Github Actions for CI/CD, standing up ECR and IAM roles with Terraform, setting up EKS to run Kubernetes jobs, managing infra monitoring with Datadog, etc., etc.”

DAGs can be published to S3 for cutting down on like half of these dependencies. And the nice thing about MWAA is log & stats publishing over cloudwatch, which should flow into any existing amazon integrated tooling.

For our team setting up terraform for iam & mwaa, some deploy pipelines to s3, and connecting some config bits to wire up splunk logs / monitoring pieces was not that much work. Initiating a separated vendor relationship & pricing out data ingress/egress costs would blow that work out of the water but maybe it’s a difference in company size/placement.

Post reply on HN