Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

ETL Pipelines with Airflow: The Good, the Bad and the Ugly

airbyte.io

Post reply on HN
Print view View original

41–50 of 87 posts

Re: ETL Pipelines with Airflow: The Good, the Bad and the Ugly

#41

Post by contravariant »

Arimbr wrote: post #2

[author of the article] My main concern about using Airflow for the EL parts is that sources and destinations are highly coupled with Airflow transfer operators (e.g. PostgresToBigQueryOperator). The community needs to provide M * N operators to cover all possible transfers. Other open-source projects like Airbyte, decouple sources from destinations, so the community only needs to contribute 2 * (M + N) connectors. A…

I mostly don't bother writing separate Operators. The only part I write are the so called Hooks (which are basically just airflow's way of defining a standard way of grabbing credentials and instantiating a session object).

After that you just write a short python function that grabs the data from one hook and pushes it to another. Which is basically the (M + N) solution you mention (I think the factor 2 is unnecessary if you've already split sources and sinks).

This approach works with anything you can connect to python. Though for particularly large datasets you want to be careful that you don't accidentally store all data in memory at once. And sure you can sometimes specialize an operation for a particular use case (e.g. if in your example can instruct BigQuery to connect to the Postgres application natively), but usually it works just fine to use a python script in-between.

Re: ETL Pipelines with Airflow: The Good, the Bad and the Ugly

#42

Post by yamrzou »

I have worked with Airflow during the past three years, but recently we adopted Dagster and I have been using it for the past 3 months. I have found it quite joyful to use and the experience has been very positive. Its main advantages compared to Airlfow (IMO):

  - A great UI
  - It forces you to clearly define inputs, outputs and types.
  - Separation of concerns: Between configuration and data, between processing and IO, and between code and deployment options.
  - It allows you to define flexible dags which you can configure at runtime, which makes it easiy to run locally or in k8s, or to switch the storage backend depending on the environment.
This blog post by the founder outlines the differences between the two in much more detail: https://dagster.io/blog/dagster-airflow

Re: ETL Pipelines with Airflow: The Good, the Bad and the Ugly

#43

Post by StreamBright »

yamrzou wrote: post #42

I have worked with Airflow during the past three years, but recently we adopted Dagster and I have been using it for the past 3 months. I have found it quite joyful to use and the experience has been very positive. Its main advantages compared to Airlfow (IMO): - A great UI - It forces you to clearly define inputs, outputs and types. - Separation of concerns: Between configuration and data, between processing and IO,…

I was looking for a replacement for Airflow right now. This is really helpful.

Re: ETL Pipelines with Airflow: The Good, the Bad and the Ugly

#45

Post by DevKoala »

Arimbr wrote: post #2

[author of the article] My main concern about using Airflow for the EL parts is that sources and destinations are highly coupled with Airflow transfer operators (e.g. PostgresToBigQueryOperator). The community needs to provide M * N operators to cover all possible transfers. Other open-source projects like Airbyte, decouple sources from destinations, so the community only needs to contribute 2 * (M + N) connectors. A…

My team is currently evaluating Debezium for a PG to S3 CDC solution, and is very advanced with the PoC.

What would be the argument for trying AirByte? It is the first time we hear about it.

Re: ETL Pipelines with Airflow: The Good, the Bad and the Ugly

#46

Post by chrisjc »

sails wrote: post #27
0x500x79 wrote: post #16

Some people have noted that this is a very Airbyte specific article, but I think that the lessons learned are still important. I have managed Airflow as a managed service for a company that has thousands of DAGs and one of our keys to success was splitting the compute and scheduling concepts into different components. We standardized on where our compute ran (Databricks, Spark, Lambdas, or K8s jobs) and had Airflow p…

How best to achieve this? I'm considering Astronomer or AWS hosted airflow, with a benefit to AWS having the compute components easily accessible within the AWS ecosystem. Starting out with a smaller scale and lower commitment to Airflow, so I'd like highest reliability with least hassle. https://aws.amazon.com/blogs/aws/introducing-amazon-managed-...

We used Astronomer at my last job. Sure beats trying to run your own instance and the Astronomer team were fantastic.

Re: ETL Pipelines with Airflow: The Good, the Bad and the Ugly

#47

Post by swyx »

> The main issue with Airflow transfer operators is that if you want to support transfers from M sources to N destinations, the community would need to code N x M Airflow operators.

I'm biased but this is a nonissue with workflow-as-code solutions like temporal.io (which Airbyte uses). N activities pulling data from sources, M activities sending data to destinations, write whatever translation layers you want in your workflows.

links to examples https://temporal.io/usecases#Pipelines and our community meetup where Airbyte spoke about their needs https://www.youtube.com/watch?v=K25Bt5asd8I

Re: ETL Pipelines with Airflow: The Good, the Bad and the Ugly

#48

Post by Arimbr »

DevKoala wrote: post #45
Arimbr wrote: post #2

[author of the article] My main concern about using Airflow for the EL parts is that sources and destinations are highly coupled with Airflow transfer operators (e.g. PostgresToBigQueryOperator). The community needs to provide M * N operators to cover all possible transfers. Other open-source projects like Airbyte, decouple sources from destinations, so the community only needs to contribute 2 * (M + N) connectors. A…

My team is currently evaluating Debezium for a PG to S3 CDC solution, and is very advanced with the PoC. What would be the argument for trying AirByte? It is the first time we hear about it.

Airbyte CDC is based on Debezium, but Airbyte abstracts it away and make it easier to CDC from Postgres, MySQL, MSSQL to any supported destination (included S3). Here is the doc for CDC: https://docs.airbyte.io/understanding-airbyte/cdc

I guess one benefit is that you can use Airbyte for all your data syncs, CDC and non-CDC. You can give it a try with your own data, and see if it's easier for your team. You can run Airbyte locally with Docker Compose: https://docs.airbyte.io/quickstart/deploy-airbyte

Re: ETL Pipelines with Airflow: The Good, the Bad and the Ugly

#49

Post by gunnarmorling »

Arimbr wrote: post #48
DevKoala wrote: post #45

Earlier quoted context omitted.

My team is currently evaluating Debezium for a PG to S3 CDC solution, and is very advanced with the PoC. What would be the argument for trying AirByte? It is the first time we hear about it.

Airbyte CDC is based on Debezium, but Airbyte abstracts it away and make it easier to CDC from Postgres, MySQL, MSSQL to any supported destination (included S3). Here is the doc for CDC: https://docs.airbyte.io/understanding-airbyte/cdc I guess one benefit is that you can use Airbyte for all your data syncs, CDC and non-CDC. You can give it a try with your own data, and see if it's easier for your team. You can run A…

Can you elaborate on how Airbyte makes things easier for a user? Would love to pick up any potential improvements in Debezium itself, so that all its users get to benefit from them, rather than only users of a specific integrator like Airbyte.

Disclaimer: I work on Debezium

Re: ETL Pipelines with Airflow: The Good, the Bad and the Ugly

#50

Post by CRConrad »

TrealTwan wrote: post #40

The article says that "SQL is taking over Python to transform and analyze data in the modern data stack". Are other people starting to notice this at ELT becomes more populate than traditional ETL? Haven't used Airflow before but use Azure Data Factory in my org to load the raw the data into the data warehouse and then transform into data models using SQL.

> Are other people starting to notice this at ELT becomes more populate than traditional ETL?

I invented ELT.

No, duh, of course not... But: I and my then-colleagues, other DW consultants, started noticing some time in the early-to-mid-00s that what we were doing didn't actually follow the order of the "ETL" acronym, and occasionally commented to each other along the lines of "shouldn't this be called 'ELT' to be more descriptive?". So I don't quite get the "starting to notice this at ELT becomes more popula[r]" bit -- this feels quite old to me. Before most of the meteoric rise of Python.

Post reply on HN