Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

ETL Pipelines with Airflow: The Good, the Bad and the Ugly

airbyte.io

Post reply on HN
Print view View original

21–30 of 87 posts

Re: ETL Pipelines with Airflow: The Good, the Bad and the Ugly

#21

Post by KptMarchewa »

rockostrich wrote: post #13
Grimm1 wrote: post #9

Earlier quoted context omitted.

In a previous job we/I trained the analysts to be more technical and write the T part of ELT in DBT. They effectively became what is known as "Analytics Engineers" so they owned the T and then wrote their analysis on top of the models they had created.

That works for ELT, especially if you have documentation around the raw data being loaded in but sounds like it adds a bit of overhead to the analysts' jobs which may or may not be more than just having the engineering team own it and document it well (something they already have to do for the analysts to write transformation code). I'm curious how you handle the upstream data schema changing. Loading in raw data mea…

In the end, analysts are cheaper than engineers.

Re: ETL Pipelines with Airflow: The Good, the Bad and the Ugly

#23

Post by rockostrich »

KptMarchewa wrote: post #21
rockostrich wrote: post #13

Earlier quoted context omitted.

That works for ELT, especially if you have documentation around the raw data being loaded in but sounds like it adds a bit of overhead to the analysts' jobs which may or may not be more than just having the engineering team own it and document it well (something they already have to do for the analysts to write transformation code). I'm curious how you handle the upstream data schema changing. Loading in raw data mea…

In the end, analysts are cheaper than engineers.

Not if it's just a part of those engineers' jobs. They're already familiar with the underlying application data so owning the transformation is just understanding what the data needs to look like and documenting it. They're going to need to document the raw data anyway to avoid those analysts asking them a million questions. Might as well avoid hiring analysts who can also learn the transformation bit and just give them good data.

Re: ETL Pipelines with Airflow: The Good, the Bad and the Ugly

#24

Post by rockostrich »

Arimbr wrote: post #17
rockostrich wrote: post #12

Earlier quoted context omitted.

Yea, I wasn't familiar with Airbyte before writing that comment so now I'm seeing the value in it. We have tons of teams asking "how do I get this data into BigQuery" and the answer is usually "use this airflow operator to dump it into GCS and then use this airflow operator to load it into BigQuery" which isn't super useful for a non-technical person or even really any technical person not familiar with Airflow. A me…

Nice work there! I also think that the next challenge for data teams is all this data documentation and discovery work. I still think that Airflow is great for power data engineers. Airbyte and dbt are positioned to empower data analysts (or lazy data engineers like me) to own the ELTs.

Agreed. I see a lot of folks coming up with one off solutions for pulling data out of 3rd party sources like Kustomer or Lever. Giving a centralized UI for setting that up would be a great service.

Seems like I have a fun weekend project.

Re: ETL Pipelines with Airflow: The Good, the Bad and the Ugly

#25

Post by wara23arish »

willvarfar wrote: post #4

Very solid article. Even with where it’s published, it’s jolly sensible. I would like to jump in and say use Beam instead of DBT, but tbh that’s bad advice. What the world needs is something open source with the incremental model of beam, a fast incremental backend (thinking htap storage that mixes columns and rows automagically) and the ease and maintainability of DBT. There is just this massive hole. If some combin…

Are you talking about Apache Beam? I happened to land a job where I had learned how to create dataflow pipelines using Apache Beam (java) on gcp.

Im a little worried that I might be investing my time on a skill/tool that isn’t that much in demand. (Instead of common backend/frontend development)

Re: ETL Pipelines with Airflow: The Good, the Bad and the Ugly

#26

Post by rockostrich »

fredliu wrote: post #20

I've never used Airflow, but used Step Function in AWS to pretty much achieved the same things this article described. I wonder if anybody has used both and what are the pros and cons between them? Besides the obvious reason of Step Function in AWS so it would work better within AWS ecosystem and Airflow is open source and service/provider agnostic?

Yea they look pretty similar. Not sure what the configuration language around step function is, but Airflow is nice because it's just python. There are a few gotchas you encounter, especially around templating, when you first get started but besides that it's pretty low overhead to start doing some pretty complex things since it's just writing python code.

Re: ETL Pipelines with Airflow: The Good, the Bad and the Ugly

#27

Post by sails »

0x500x79 wrote: post #16

Some people have noted that this is a very Airbyte specific article, but I think that the lessons learned are still important. I have managed Airflow as a managed service for a company that has thousands of DAGs and one of our keys to success was splitting the compute and scheduling concepts into different components. We standardized on where our compute ran (Databricks, Spark, Lambdas, or K8s jobs) and had Airflow p…

How best to achieve this? I'm considering Astronomer or AWS hosted airflow, with a benefit to AWS having the compute components easily accessible within the AWS ecosystem.

Starting out with a smaller scale and lower commitment to Airflow, so I'd like highest reliability with least hassle.

https://aws.amazon.com/blogs/aws/introducing-amazon-managed-...

Re: ETL Pipelines with Airflow: The Good, the Bad and the Ugly

#28

Post by higeorge13 »

fredliu wrote: post #20

I've never used Airflow, but used Step Function in AWS to pretty much achieved the same things this article described. I wonder if anybody has used both and what are the pros and cons between them? Besides the obvious reason of Step Function in AWS so it would work better within AWS ecosystem and Airflow is open source and service/provider agnostic?

A couple points from my end:

- I miss the airflow ui to monitor the workflow execution, clear failed tasks, its pre built notifications, emails and so on.

- Passing state between step functions is a bit tricky tbh. I found airflow’s way a little more straightforward.

- Step functions seem to scale better, especially when you aim to use it for dynamic workflows. The recent update of step functions where you can literally use any aws service api (compared to a couple weeks ago where you were restricted to around 10 and then you only had lambdas) minimises its weaknesses imo.

Re: ETL Pipelines with Airflow: The Good, the Bad and the Ugly

#29

Post by higeorge13 »

mkw5053 wrote: post #19

Having always been on AWS and using Glue Spark Jobs for my jobs, I've never felt any benefit of using Airflow for orchestration over Glue Workflows. I can understand some people not wanting to deal with vendor lock-in. I'm curious what others opinions are.

For glue jobs you probably don’t need something more. However step functions is the norm for creating workflows in aws nowadays.

Re: ETL Pipelines with Airflow: The Good, the Bad and the Ugly

#30

Post by numlocked »

sails wrote: post #27
0x500x79 wrote: post #16

Some people have noted that this is a very Airbyte specific article, but I think that the lessons learned are still important. I have managed Airflow as a managed service for a company that has thousands of DAGs and one of our keys to success was splitting the compute and scheduling concepts into different components. We standardized on where our compute ran (Databricks, Spark, Lambdas, or K8s jobs) and had Airflow p…

How best to achieve this? I'm considering Astronomer or AWS hosted airflow, with a benefit to AWS having the compute components easily accessible within the AWS ecosystem. Starting out with a smaller scale and lower commitment to Airflow, so I'd like highest reliability with least hassle. https://aws.amazon.com/blogs/aws/introducing-amazon-managed-...

At Grove we make extensive use of KubernetesPodOperator[0]. This allows to both encapsulate the job in containers (k8s pods), and also specify the resources we need for each DAG.

[0] https://airflow.apache.org/docs/apache-airflow-providers-cnc...

Post reply on HN