Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Airflow's Problem

stkbailey.substack.com

Post reply on HN
Print view View original

91–100 of 126 posts

Re: Airflow's Problem

#91

Post by glogla »

shmoogy wrote: post #65
glogla wrote: post #16

Earlier quoted context omitted.

If it is just you, you are fine, and I'm not sure other tools would have that much benefit. Trouble with Airflow starts when multiple teams and user types start to share it.

I've definitely noticed more issues after adding users, but it's more that they don't actually understand a lot of what they're trying to do and cause problems when writing dags.

Yeah, Airflow isn't multi-tenant.

People can potentially overwrite each other's DAGs. Credential management is complicated. Broken DAG can stop whole Airflow. Slow DAG can impact performance of whole Airflow. Getting DAGs to wait for each other (like one team prepares data up to a point and then other team builds on that) is kind of a nightmare. Sometimes people want features from newer Airflow, but some other team built DAG that isn't forward compatible. Etc etc.

But I'm not sure there actually is a better solution elsewhere. At least I have not seen it yet, maybe Dagster is on a good road.

But as I said, for centralized solutions it works really well.

Re: Airflow's Problem

#92

Post by zentrus »

Airflow helped my team out a lot a couple years ago mainly for the simplicity of the topdown UI-based view of a complicated ETL AND the ability to retry parts of the ETL.

We had lots of lessons learned. For instance, why does PythonOperator even exist? It takes a callable and thus you're likely not going to see good coding pattern emerge for something that needs to be 1000+ LoC. Instead, we just subclassed BaseOperator and used tried-and-true OO principles.

Re: Airflow's Problem

#93

Post by tchiotludo »

All the issues described in this post lead me to create Kestra [0] . Airflow was a true revolution when it was open-source and we need thanks its innovation. But I totally agree that a large static dag is not appropriate in the actual data world with data mesh and domain responsibility.

[0] https://github.com/kestra-io/kestra

Re: Airflow's Problem

#94

Post by robertlagrant »

orthoxerox wrote: post #62

A few years ago a new guy at our DWH team tried to sell Airflow to the rest of the team. They invited me to listen to his talk as well, and I was baffled why something so trivial as Airflow was being sold as a critically important piece of infrastructure. Why would I need a glorified server-side crontab if something like MS DTS from 1998 could do the same, but better? Sure, Python is probably better than whatever DTS…

> isn't Airflow's problem, it's your problem

This is a baffling statement.

Re: Airflow's Problem

#95

Post by mumblemumble »

systemvoltage wrote: post #86
mumblemumble wrote: post #44

Earlier quoted context omitted.

With all credit due to Google's excellent and under-appreciated paper Machine Learning: The High Interest Credit Card of Technical Debt [1], I submit that Big Data is the high interest home equity line of credit of business operations debt. It's not that big data tools aren't useful. It's that, when you just start amassing huge piles of data without a clear up-front plan for how it will be used, and assume that a who…

Yet, we abide by data-oriented conclusions outside of software engineering all the time. From Academics papers to FDA to crime statistics.

I won't say any of those are perfect. But there's at least a little more effort toward responsible data analysis in academia. The FDA brings an interesting example to mind. Take a look at how, on paper, drugs suddenly magically became less effective when the FDA started requiring clinical trial pre-registration in 2007.

It's also worth noting that, over the past few decades, most academic fields have been getting increasingly skeptical of the value of correlative research on pre-existing data sets. Even among people who have been extensively trained in how to do it properly. And yet, the vast majority of big data business plans I've seen in practice boil down to "collect a huge data set and then let people do correlative research on it."

Re: Airflow's Problem

#96

Post by htahir111 »

Wouldn't it make sense to decouple the orchestration later from the authoring layer for the dags? That way you could solve the authoring problems separately from the lower level orchestration problems. We're trying this over at ZenML (https://zenml.io) but have yet to get feedback

Re: Airflow's Problem

#97

Post by systemvoltage »

mumblemumble wrote: post #95
systemvoltage wrote: post #86

Earlier quoted context omitted.

Yet, we abide by data-oriented conclusions outside of software engineering all the time. From Academics papers to FDA to crime statistics.

I won't say any of those are perfect. But there's at least a little more effort toward responsible data analysis in academia. The FDA brings an interesting example to mind. Take a look at how, on paper, drugs suddenly magically became less effective when the FDA started requiring clinical trial pre-registration in 2007. It's also worth noting that, over the past few decades, most academic fields have been getting inc…

Agreed, I want more scrutiny than some entity flashing “Here is the data”. It can easily be exploited behind the veneer of data-based-credibility.

Re: Airflow's Problem

#98

Post by mason55 »

itsoktocry wrote: post #34
llambda wrote: post #9

To address a point the author makes: I’m entirely unconvinced the “shift left” mentality of data democracy (aka business operators should write sql) is actually shifting left or a worthy path to pursue for most businesses. More recently this 2010s fad seems to be dying and in favor we’re seeing centralized data efforts that produce data products. One of the most significant pitfalls of data is failing to interrogate…

> that if you give everyone access all the time the magic will happen There's much ongoing discussion about this is the data world, often revolving around "self-service analytics". Unless you're talking about "our analysts don't have to clean data all the time", which, for a large enough organization makes sense, "self-service" for non-technical folks is futile and pointless. They need specific answers to specific qu…

This is the constant argument I have with people about data products.

You don't need to expose more dimensions or get the users more access to the raw data. You need to understand what their business is and what their business problems are and help them answer those specific questions quickly and succinctly.

Yes, there are certainly times where people use huge amounts of raw data to uncover the answer to a question they didn't know they had. But it's rare, it's expensive to support, and most businesses are going to be able to do anything with it anyway (a whole org built to do X isn't suddenly going to shift to do Y because you discovered some insight in a random report).

Re: Airflow's Problem

#99

Post by ricklamers »

jon_adler wrote: post #43

Is there any love for the Argo [1] project suite (Workflows, Events, CD) for this type of use case? I haven’t tried it out myself yet however it does look interesting. [1] https://argoproj.github.io

Argo is pretty amazing if you want to take advantage of the work Kubernetes has done to scale resource efficiently across a cluster of compute nodes.

If you’re looking for something that’s a bit more high level and friendly to expose directly to your data team (data scientists/data engineers/data analysts) you can check out https://github.com/orchest/orchest

You can think of it as a browser UI/workbench for Argo scheduled pipelines. Disclaimer: author of the project

Re: Airflow's Problem

#100

Post by ricklamers »

dominotw wrote: post #26

Just put everything in a warehouse( via fivetran or some such thing) and just use DBT. Use airflow as cron runner for dbt. If you don't need realtime metrics, this formula works way better than convoluted airflow dags.

We have had a great experience scheduling Meltano/dbt inside Orchest for our Metabase dashboards. As a pattern, combining these declarative/configuration CLI tools with a flexible orchestration layer (Orchest can run any containerized task, it will containerize transparently for you) really shines.
Post reply on HN