Earlier quoted context omitted.
Nice work there! I also think that the next challenge for data teams is all this data documentation and discovery work. I still think that Airflow is great for power data engineers. Airbyte and dbt are positioned to empower data analysts (or lazy data engineers like me) to own the ELTs.
Agreed. I see a lot of folks coming up with one off solutions for pulling data out of 3rd party sources like Kustomer or Lever. Giving a centralized UI for setting that up would be a great service. Seems like I have a fun weekend project.
ETL Pipelines with Airflow: The Good, the Bad and the Ugly
71–80 of 87 posts
Re: ETL Pipelines with Airflow: The Good, the Bad and the Ugly
#72Very solid article. Even with where it’s published, it’s jolly sensible. I would like to jump in and say use Beam instead of DBT, but tbh that’s bad advice. What the world needs is something open source with the incremental model of beam, a fast incremental backend (thinking htap storage that mixes columns and rows automagically) and the ease and maintainability of DBT. There is just this massive hole. If some combin…
Are you perhaps talking about something like https://materialize.com/ ? (btw, dbt now has some materialize compatibility) Maybe Pravega and Beam working together? https://pravega.io/docs/v0.6.0/key-features/ Another option is something like Snowflake with tasks and streams. https://docs.snowflake.com/en/user-guide/tasks-intro.html Or Snowflake with change streams, dbt and scheduler in combination with lambda views. h…
>Just don’t do it. Because dbt is primarily designed for batch-based data processing, you should not schedule your dbt jobs to run continuously. This can open the door to unforeseeable bugs.
why not though. you can inplement incremental models and run them continously. sure its more work but what bugs does this cause?
Re: ETL Pipelines with Airflow: The Good, the Bad and the Ugly
#73I recently started working on my own DAG execution framework, after failing to get some patches into Airflow to make the scheduling easier to reason about. My typical use case was orchestrating DAGs with thousands of vertices, and airflow would silently wedge itself and fail to report errors. Daggy is just starting out, but I’m hoping it’ll become more robust and scalable as time goes on. https://gitlab.com/iroddis/d…
(workflow/DAG systems are an area of special interest for me...) Happy to see you're supporting runtime-dynamic DAG shape. That's one of the critical things Airflow is missing. But, if I'm inferring correctly, it looks like you're recording task state in a similar way to Airflow, i.e. in some kind of data store. I suggest you look at how Luigi models task state: using "targets" which are a task's materialized outputs…
Build systems such as Bazel are build on the same incremental DAG processing foundation.
I agree that it’s the most important feature, as it dramatically improves correctness and performance (when at scale).
IAM policy trouble is a good example of a bug that crashes a pipeline but has no influence on precomputed outputs of succeeded nodes.
Re: ETL Pipelines with Airflow: The Good, the Bad and the Ugly
#74Something I've been thinking is, like the article says, SQL is a very good way to transform data, and it recommends dbt for it, but how to you test this transformation? I know dbt has tests, but on a superficial look they seem pretty trivial stuff like "check if this field is null" and things like that, but what about tests which I setup scenarios and see if the end result of my transformation is what I expect? Is th…
Checkout great expectations ( https://greatexpectations.io/ ). You basically run assertions on your data. It auto-generates documentation for you and you can also store results of your validations in a flat file or database. I think there are dbt modules that try to mimic great expectations' validation but I like the docs that come with great expectations
Check out dbt-expectations package[1]. It's a port of the Great Expectations checks to dbt as tests. The advantage of this is you don't need another tool for these pretty standard tests, and can be early incorporated into dbt workflows.
Re: ETL Pipelines with Airflow: The Good, the Bad and the Ugly
#75Something I've been thinking is, like the article says, SQL is a very good way to transform data, and it recommends dbt for it, but how to you test this transformation? I know dbt has tests, but on a superficial look they seem pretty trivial stuff like "check if this field is null" and things like that, but what about tests which I setup scenarios and see if the end result of my transformation is what I expect? Is th…
You maintain pipes that can route water, oil or gasoline. You don't want to test for water purity, because next day you are asked to route sewage or oil through your system. You can at best test volume, pressure or velocity in the pipeline because these actually have an impact on your system.
The actual test that business is asking you to put at the "raw data" stage has to come at the application level later on. Eg. when you extract a metric out of the data you can do all kinds of time series tests on it, which will test the pipeline thoroughly.
Testing is what makes data engineering different from software engineering. You don't control the input data and don't want to. You have to make your organization work in such a way that downstream users communicate data issues back to you. You only test the data thoroughly by using it.
If you are a data vendor and your downstream user is a paying customer that you don't want to embarass yourself in front of, you need to invent a use for the data, that you sell, within your org.
Re: ETL Pipelines with Airflow: The Good, the Bad and the Ugly
#76I recently started working on my own DAG execution framework, after failing to get some patches into Airflow to make the scheduling easier to reason about. My typical use case was orchestrating DAGs with thousands of vertices, and airflow would silently wedge itself and fail to report errors. Daggy is just starting out, but I’m hoping it’ll become more robust and scalable as time goes on. https://gitlab.com/iroddis/d…
(workflow/DAG systems are an area of special interest for me...) Happy to see you're supporting runtime-dynamic DAG shape. That's one of the critical things Airflow is missing. But, if I'm inferring correctly, it looks like you're recording task state in a similar way to Airflow, i.e. in some kind of data store. I suggest you look at how Luigi models task state: using "targets" which are a task's materialized outputs…
> resuming a partially failed build
Daggy does this! Right now it will continue running the DAG until every path is completed or all vertices in a processing state (queued, running, retry, error) are in the error state, then the DAG goes to an error state.
It's possible to explicitly set task/vertex states (e.g. mark it complete if the step was manually completed), then change the DAG state to QUEUED, at which point the DAG will resume execution from where it left off. [1] is a unit test that walks through that functionality.
[1] https://gitlab.com/iroddis/daggy/-/blob/master/tests/unit_se...
Re: ETL Pipelines with Airflow: The Good, the Bad and the Ugly
#77Has anyone kicked the tires on Airflow, Prefect, and Dagster and care to give their thoughts? My initial foray into Airflow 1 met a lot of complexity that both Prefect and Dagster claim to minimize.
Tried some of them, threw them away for various reasons and used Argo Workflows instead. Airflow mixes orchestration and the actual transformation, and I wasn’t a fan of being ao intimately tied to Python. Argo let’s us run arbitrary containers (a mix of Spark, Rust, etc) and played nicer with the tools we were already using (Kafka, K8s etc), didn’t require us to write Python and cleanly separated orchestration and a…
Re: ETL Pipelines with Airflow: The Good, the Bad and the Ugly
#78Earlier quoted context omitted.
+1 In my org we never use Airflow to compute anything. We only use a single operator: PythonOperator. All business and data access logic is encapsuled in REST APIs. Our DAGs are responsible for calling those APIs in the correct order, in the correct time, retrying when needed. I really dislike using Airflow for anything else, as it generally becomes a huge mess.
If it's all Python, why not use Prefect?
Re: ETL Pipelines with Airflow: The Good, the Bad and the Ugly
#79[author of the article] My main concern about using Airflow for the EL parts is that sources and destinations are highly coupled with Airflow transfer operators (e.g. PostgresToBigQueryOperator). The community needs to provide M * N operators to cover all possible transfers. Other open-source projects like Airbyte, decouple sources from destinations, so the community only needs to contribute 2 * (M + N) connectors. A…
1 - Hide your ETL logic in REST APIs and use common microservice patterns for monitoring, logging, etc... 2 - Use PythonOperator to call your APIs. It's working really well for us.
Re: ETL Pipelines with Airflow: The Good, the Bad and the Ugly
#80 B ) Lack of granular roles and security groups which leads to rely on trust that no airflow users mistakenly make any changes through UI
I feel there is some undocumented dependency between scheduler, celery and web server which always hit performance issue of ETL job.Also We see more reliability issues on the platform as more workloads are added.