51–60 of 87 posts
I know dbt has tests, but on a superficial look they seem pretty trivial stuff like "check if this field is null" and things like that, but what about tests which I setup scenarios and see if the end result of my transformation is what I expect? Is there any good tools for this?
Very solid article. Even with where it’s published, it’s jolly sensible. I would like to jump in and say use Beam instead of DBT, but tbh that’s bad advice. What the world needs is something open source with the incremental model of beam, a fast incremental backend (thinking htap storage that mixes columns and rows automagically) and the ease and maintainability of DBT. There is just this massive hole. If some combin…
Are you talking about Apache Beam? I happened to land a job where I had learned how to create dataflow pipelines using Apache Beam (java) on gcp. Im a little worried that I might be investing my time on a skill/tool that isn’t that much in demand. (Instead of common backend/frontend development)
Something I've been thinking is, like the article says, SQL is a very good way to transform data, and it recommends dbt for it, but how to you test this transformation? I know dbt has tests, but on a superficial look they seem pretty trivial stuff like "check if this field is null" and things like that, but what about tests which I setup scenarios and see if the end result of my transformation is what I expect? Is th…
My typical use case was orchestrating DAGs with thousands of vertices, and airflow would silently wedge itself and fail to report errors.
Daggy is just starting out, but I’m hoping it’ll become more robust and scalable as time goes on.
Some people have noted that this is a very Airbyte specific article, but I think that the lessons learned are still important. I have managed Airflow as a managed service for a company that has thousands of DAGs and one of our keys to success was splitting the compute and scheduling concepts into different components. We standardized on where our compute ran (Databricks, Spark, Lambdas, or K8s jobs) and had Airflow p…
+1 In my org we never use Airflow to compute anything. We only use a single operator: PythonOperator. All business and data access logic is encapsuled in REST APIs. Our DAGs are responsible for calling those APIs in the correct order, in the correct time, retrying when needed. I really dislike using Airflow for anything else, as it generally becomes a huge mess.
[author of the article] My main concern about using Airflow for the EL parts is that sources and destinations are highly coupled with Airflow transfer operators (e.g. PostgresToBigQueryOperator). The community needs to provide M * N operators to cover all possible transfers. Other open-source projects like Airbyte, decouple sources from destinations, so the community only needs to contribute 2 * (M + N) connectors. A…
1 - Hide your ETL logic in REST APIs and use common microservice patterns for monitoring, logging, etc... 2 - Use PythonOperator to call your APIs. It's working really well for us.
The article says that "SQL is taking over Python to transform and analyze data in the modern data stack". Are other people starting to notice this at ELT becomes more populate than traditional ETL? Haven't used Airflow before but use Azure Data Factory in my org to load the raw the data into the data warehouse and then transform into data models using SQL.
Use Jenkins. You'll be happy you did.
Blows my mind that they recommend keeping Airflow just for the scheduler , which is basically the WORST part of an overall bad tool. Use Jenkins. You'll be happy you did.
If I'm not mistaken, Jenkins directly tells you not to use it as a scheduler