I don't really agree with decentralized ETL. You can't imagine the mess people proudly wrote. And yeah I have been on the other aide of the trench too.
Airflow's Problem
71–80 of 126 posts
Re: Airflow's Problem
#72I don’t understand the desire to describe DAGs in Python… it’s a fine scripting language but pretty horrible for this declarative description stuff.
Re: Airflow's Problem
#73This is why I though the shift from "orchestrate jobs" to "keep track of state of assets" that Dagster is trying to do is pretty important. But it sems it might not be enough - it still keeps clunky (pythonic) interface and I don't know how well it does multi-tenancy.
Could you clarify what "keep track of state of assets" means?
It's a mindset shift to a more declarative model. The idea has also popped up in other niche orchestrators.
This is an oversimplification but IMO the easiest way of picturing it is instead thinking of defining your graph as a forward moving thing w/ the orchestrator telling things they can be run you shift to defining your graph nodes to know their dependencies and they let the orchestrator know when they're runnable.
Re: Airflow's Problem
#74The problem with airflow is its scalability (or lack thereof) and DAG dependency management.
Airflow successors must figure out how to distribute the cron and all dependencies should be self contained in a Docker image.
Re: Airflow's Problem
#75The post feels like a bait-and-switch in the sense that it presents itself as about Airflow's shortcomings but focuses mostly on problems that Airflow doesn't attempt to solve. Airflow can certainly be frustrating and it doesn't solve _all_ workflow orchestration problems. Surely the same thing can be said of many tools? This seems mostly like a mismatch of expectations.
When we engage in complex work, it is important to keep our options open so we can direct our best effort to the hardest problems.
It is rarely clear what the hard problems will be when new to a domain. Only as scale kicks in.
We are constantly pitched frameworks that sell themselves as a good approach to a domain, but then obstruct engagement with the hardest problems when it matters. The developer becomes captive of the system that claimed it would steer them right.
This is particularly true of fields where the hard problems are integration problems which, by their nature, cannot be outsourced to frameworks.
Re: Airflow's Problem
#76Earlier quoted context omitted.
I think context matters and the title "Airflow's Problem" doesn't make much sense when talking about the physical phenomenon of flowing air.
Not really, the past two years have seen probably a order of magnitude increase in interest and articles about industrial-scale ventilation and air quality.
You still wouldn't phrase it like "Airflow's Problem" because the concept of airflow is incapable of having a problem. It just exists. Some theories _about_ airflow may have problems, but then you would specify that in the title.
Re: Airflow's Problem
#77ETL seems just like one of those perennial challenges that resist humanity's efforts to categorize the world into need and tidy boxes
Re: Airflow's Problem
#78Recent perspectives from the creators of Prefect, Dagster, Flyte, and Orchest
=> https://gradientflow.com/summer-of-orchestation/
Re: Airflow's Problem
#79Earlier quoted context omitted.
I'd suggest to raise the issue in the devlist or on GitHub Issues to get more visibility.
I'd be okay with the article headline on HN being "Apache Airflow's Problem" so that I know it's about a piece of apache software and not something interesting about airflow.
The Apache Software Foundation would also prefer this, per their trademark policy: https://www.apache.org/foundation/marks/faq/#guide
Re: Airflow's Problem
#80Dismissing Airflow for not being Astronomer is like dismissing Linux for not having the capabilities of a large-scale hypervisor. Replace “Airflow” with “Linux,” “data engineers” with “systems programmers,” and “Astronomer” with your hypervisor of choice (Xen/VMWare/etc.), and you can see how absurd the author’s point is: My problem is that ~Airflow~ Linux was not designed to address [high-level systems architecture]…
Managed Airflow doesn't even solve any of the author's outlined frustrations. It keeps the "obscene" syntax, it's still stateless, it's not "decentralized" etc. Honestly, the article is so disingenuous that it comes off like a paid-for puff piece for Astronomer. It's the article-equivalent of the late-night infomercial guy who rips open a bag of potato chips like the hulk because he doesn't have this special tool tha…
FYI, the infomercials with the strange tools fixing strange problems are usually focused on old or disabled people. Opening bag of chips with ridiculius tool sounds stupid, but it might help a stroke survivor or someone with one arm - but the sellers don't want to show those struggle on the screen to avoid humiliating people, so you see pefectly healthy looking young people spilling things like they have some neurodegenerative disorder or something. Because the target audience might.
Not saying infomercials people are angels, of course, but I wanted to sharethus somewhat nonobvious context.
(To stretch the metaphor, Airflow management system that gives everyone their own Airflow might be ridiculous but make sense for companies where cooperation is difficult :))