Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Using an ETL framework vs. writing yet another ETL script

airbyte.io

Post reply on HN
Print view View original

21–30 of 77 posts

Re: Using an ETL framework vs. writing yet another ETL script

#21

Post by davinchia »

lloydatkinson wrote: post #14
mrbungie wrote: post #12

We currently are using Airflow for ELTs/ETLs to ingest from different Postgres databases to BigQuery/Google Cloud Storage. Airbyte looks sweet for the same task and would free us from a big effort burden, but its Postgres source only supports SELECT * statements (i.e. you can't deselect columns). That's kind of a dealbreaker for us, because for security reasons our Postgres users permissions are granularly configured…

That's crazy, why would it have the approach of "select all columns" when you might never need all of them? Could you create a view for it to select from instead? Either way if someone told me a ETL supports only literally all the columns or nothing then I'd assume it was a MVP product to demonstrate the idea... not a production system.

We are working on it. This should be out next quarter!

Re: Using an ETL framework vs. writing yet another ETL script

#22

Post by Cwizard »

elchief wrote: post #10

I gave up on all these frameworks and just use Cron and Make and Python

Do you use some kind of library to help with transformations? If you have some volume using plain python can become a bottleneck no? Any tips in this area would be appreciated.

Re: Using an ETL framework vs. writing yet another ETL script

#24

Post by WhyCause »

Cwizard wrote: post #22
elchief wrote: post #10

I gave up on all these frameworks and just use Cron and Make and Python

Do you use some kind of library to help with transformations? If you have some volume using plain python can become a bottleneck no? Any tips in this area would be appreciated.

I'm not the person you were asking, but I, too, use mostly Python for ETL tasks as a good chunk of my job.

Python's speed, or lack thereof, is rarely an issue. Most tasks that I write/run finish in less than a minute, including the time for transformations. Some of those can get fairly hairy, as well. Most of the time is spent querying the DB and uploading to the remote.

For the tasks where the transformations are too complex, you can greatly decrease the time necessary by tweaking the queries (i.e., limiting rows to just changed rows, etc.). And frankly, once you realize most data doesn't need to update in real time, it doesn't matter how long the transformation step takes (as long as it's fewer than about 23 1/2 hours).

Re: Using an ETL framework vs. writing yet another ETL script

#25

Post by computershit »

I have been working a lot in this space for the last two years but especially in the last 6 months. I believe we're about to enter a phase where much more elegant and less restrictive ETL platforms or frameworks are as commonplace as modern software CICD offerings. Prefect and Dagster both stand out to me as viable replacements for Airflow.

Re: Using an ETL framework vs. writing yet another ETL script

#26

Post by elchief »

Cwizard wrote: post #22
elchief wrote: post #10

I gave up on all these frameworks and just use Cron and Make and Python

Do you use some kind of library to help with transformations? If you have some volume using plain python can become a bottleneck no? Any tips in this area would be appreciated.

pyspark if big-ol' data sets

pandas otherwise

Re: Using an ETL framework vs. writing yet another ETL script

#28

Post by johanneskanybal »

cheriot wrote: post #23

Can any of these ETL frameworks kick off an ETL script without a rewrite? Something that would handle scheduling, retries, emit metrics around those actions, but let me use my own tools for the actual data manipulation.

One of the standard patterns is airflow and kubernetes executor and then the pod can do whatever. There's the occasional side effect and some setup but more or less it works as advertised.
Post reply on HN