21–30 of 126 posts
Earlier quoted context omitted.
So to confirm, you are annoyed that the name of this project is a word.
Definitely. I'd like the name of this project to be changed to be a name, not a common word. Thanks.
I agree airflow is old, legacy and ideally folks should not use it, reality is there is a lot of pipelines already built with it - sadly. I think as a community we have to start moving away from it for more complicated problems.
Disclaimer: I created Flyte.org and heavily believe in decentralized development of DAGs and centralized management of infrastructure
On Dagster and Prefect you communicate between tasks as if you were writing pure Python. On Airflow on the other hand ...
More recent tools such as Dagster and Prefect have much more to offer. One simple example is communication between tasks. Airflow has a clunky system for that called XCom. The actual author of XCom says you probably should not use it due to the level of hackery it has under the hood [1]: On Dagster and Prefect you communicate between tasks as if you were writing pure Python. On Airflow on the other hand ... [1] https…
Use airflow as cron runner for dbt.
If you don't need realtime metrics, this formula works way better than convoluted airflow dags.
Earlier quoted context omitted.
Definitely. I'd like the name of this project to be changed to be a name, not a common word. Thanks.
I'd suggest to raise the issue in the devlist or on GitHub Issues to get more visibility.
I've really enjoyed using taskflow (https://github.com/taskflow/taskflow) it allows us to employ our existing logging and deployment paradigms.
> Shift 1: “We know the lineage” to “We know what in god’s name is happening” Bro I can't even get my company to the _first_ part, and we're collectively already having issues with the second? What is everyone else's read on this situation in general? Do you all have row and table level lineages for your data? For pipelines that people are actively using? Every company I've ever been in can hardly figure out where fi…
In life sciences research to support synthetic control arms, the FDA is caring more about the lineage/manipulation of the data than the data science models used to predict X/Y/Z.
IE - what was the data originally, what did it end up as prior to ingestion into AIML, why was it changed, what steps were involved, etc.
There are not a ton of good out of the box solutions for data lineage and its driving me nuts.
We have Apache NIFI which promises data lineage out of the box and _appears_ to deliver. I've never implemented it though.
We have pachyderm which has some support here but I don't know about it.
Besides that it appears roll-your-own.
I kind of wish there was an accepted best practice for data lineage but its - surprisingly - wild west. And its completely 100% required for industry use.
In particular I transpile to Airflow code (can also deploy to Lambda) because I think it's still the most robust and well supported "runtime", I just don't think the developer experience is that good.