Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Airflow's Problem

stkbailey.substack.com

Post reply on HN
Print view View original

1–10 of 126 posts

Re: Airflow's Problem

#2

Post by geertj »

We tried to set up Airflow in our team in the past. The big problem we encounrted is that its unit of management (I believe it's called a "job" but I'm rusty on this) is too low level. Our pipeline processes a lot of data and we have millions of jobs per day. Once Airflow has an (planned or unplanned) outage, 10s of thousands of job start piling up, and it never recovers from that.

In the end we replaced our data orchestration with a stateless lambda that for a configured time interval 1/ looks at what output data is missing, 2/ cross-references that with running jobs (in AWS Batch), and 3/ submit jobs for missing data that has no job. Jobs themselves are essentially stateless. They are never restarted and we don't even look at their status. If one fails we notice because there will be a hole in the output and we therefore submit a new one. Some safety precautions are added to prevent a job from repeatedly failing, but that's the exception.

Maybe Airflow has moved on from when we last tried it. But this was our experience.

Re: Airflow's Problem

#4

Post by ciguy »

My current company has been having a lot of success with Dagster. It seems to give a lot more flexibility thanAirflow in terms of defining the pipeline and where to run it. It's also a bit friendlier when things fail and need to be backfilled or retried IMO. Airflow feels like it's somewhat legacy at this point in time. It served a need well but the needs have changed now.

Re: Airflow's Problem

#5

Post by glogla »

This is why I though the shift from "orchestrate jobs" to "keep track of state of assets" that Dagster is trying to do is pretty important. But it sems it might not be enough - it still keeps clunky (pythonic) interface and I don't know how well it does multi-tenancy.

Re: Airflow's Problem

#7

Post by MontyCarloHall »

Dismissing Airflow for not being Astronomer is like dismissing Linux for not having the capabilities of a large-scale hypervisor.

Replace “Airflow” with “Linux,” “data engineers” with “systems programmers,” and “Astronomer” with your hypervisor of choice (Xen/VMWare/etc.), and you can see how absurd the author’s point is:

   My problem is that ~Airflow~ Linux was not designed to address [high-level systems architecture] problems. We don’t need a better [Linux], but we need a higher-level one: a system that enables ~data engineers~ systems programmers to think at a platform level.

   In fact, [Linux] is already displaced. [Linux] qua [Linux] is already obsolete, and it happened right within the [Linux] ecosystem. It’s called ~Astronomer~ Xen/VMWare/etc.

  If it sounds like you could simply replace [Linux] with basically any other ~job execution engine~ operating system, that’s because you could.
This is where the argument falls apart. Yes, for very large, complex deployments, higher-level orchestration is important, but the choice of low-level execution engine is also still hugely relevant, just as the choice of guest OS is still hugely relevant when discussing large deployments of VMs.

Furthermore, very few people actually need very large scale deployments; user experience and capabilities at the low-level are what most users actually care about.

Re: Airflow's Problem

#9

Post by llambda »

To address a point the author makes: I’m entirely unconvinced the “shift left” mentality of data democracy (aka business operators should write sql) is actually shifting left or a worthy path to pursue for most businesses. More recently this 2010s fad seems to be dying and in favor we’re seeing centralized data efforts that produce data products.

One of the most significant pitfalls of data is failing to interrogate the value it provides and assuming that if you give everyone access all the time the magic will happen. The truth is value does not simply materialize just as value does not magically spring from computers by a human powering it on (okay sure, you may have already automated the value but that’s actually the point I’m about to make). In both cases it requires an experienced practitioner who collaborates with a larger team to intersect their work with the business needs.

Data is tricky, all the more so because it’s often seen as a panacea by business leaders who aren’t connected with the work of extracting that value.

Post reply on HN