Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Using an ETL framework vs. writing yet another ETL script

airbyte.io

Post reply on HN
Print view View original

61–70 of 77 posts

Re: Using an ETL framework vs. writing yet another ETL script

#61

Post by banku_brougham »

idiocrat wrote: post #58
travisgriggs wrote: post #11

ETL was actually a new acronym for me: Extract, Transform, Load. https://en.m.wikipedia.org/wiki/Extract,_transform,_load

Please be aware there is also a newer concept, which is ELT. Extract Load Transform. This is the data lake principle: you firstly dump all data verbatim into one coherent data storage. (data ingestion phase) The transform will eventually be done by the data scientists (data curation phase). The data curation is really difficult as it requires in-depth knowledge of the data, removing of slack, creation and lookup of r…

1. Great explanation of ELT, for a long time ETL monkey this broke through for me. 2. GDPR/COPPA means you really have to think about your data lake before dumping verbatim into long term storage.

Re: Using an ETL framework vs. writing yet another ETL script

#62

Post by banku_brougham »

elchief wrote: post #36
travisgriggs wrote: post #11

ETL was actually a new acronym for me: Extract, Transform, Load. https://en.m.wikipedia.org/wiki/Extract,_transform,_load

It really stands for Excel Taking Longtime

wow. i laughed and laughed. oh well i too am a man of culture.

Re: Using an ETL framework vs. writing yet another ETL script

#63

Post by banku_brougham »

mrbungie wrote: post #12

We currently are using Airflow for ELTs/ETLs to ingest from different Postgres databases to BigQuery/Google Cloud Storage. Airbyte looks sweet for the same task and would free us from a big effort burden, but its Postgres source only supports SELECT * statements (i.e. you can't deselect columns). That's kind of a dealbreaker for us, because for security reasons our Postgres users permissions are granularly configured…

Thats crazy. Lets see what they to do to fill out this area, somebody has surely filed an issue on the girhub repo be sure to upvote it.

Re: Using an ETL framework vs. writing yet another ETL script

#64

Post by banku_brougham »

lcuff wrote: post #54

There's a despair that overcame me when the data we were processing changed date/time format in the middle of a file. It was a file for one day's data! Data on 5 minute intervals for 3 thousand data sources. Received in an email, no less. (Long story). But date/time format changed. The initial load was garbage. As many know, some countries use mm/dd/yy and others dd/mm/yy. I just wanted to cry when I dug in and saw t…

People, no matter how skanky the script it is essential to propery orchestrate them. (I personally like Prefect).

This means investing timebin learning a workflow orchestrator, but your new hires will be glad they did.

Re: Using an ETL framework vs. writing yet another ETL script

#66

Post by jpgvm »

banku_brougham wrote: post #61
idiocrat wrote: post #58

Earlier quoted context omitted.

Please be aware there is also a newer concept, which is ELT. Extract Load Transform. This is the data lake principle: you firstly dump all data verbatim into one coherent data storage. (data ingestion phase) The transform will eventually be done by the data scientists (data curation phase). The data curation is really difficult as it requires in-depth knowledge of the data, removing of slack, creation and lookup of r…

1. Great explanation of ELT, for a long time ETL monkey this broke through for me. 2. GDPR/COPPA means you really have to think about your data lake before dumping verbatim into long term storage.

Ultimately regulations will change over time. I think the most important thing to keep in mind is storing data in a form that is amenable to large scale batch transformation. This way you can respond by kicking off a task to "rewrite the world" to scrub fields or drop records. Bug reason why I encourage stuff like Avro, Parquet or the even just loading to BigQuery.

Re: Using an ETL framework vs. writing yet another ETL script

#67

Post by wodenokoto »

davinchia wrote: post #16

Airbyte Engineer here. I think some of the points made here about ETL scripts being just 'ETL scripts' are very relevant. Definitely been on the other side of the table arguing for a quick 3-hour script. Having written plenty of ETL scripts - in Java with Hadoop/Spark, Python with Airflow and pure Bash - that later morphed into tech debt monsters, I think many people underestimate how quickly these can quickly snowba…

Where do you stand between airflow and dataflow?

Re: Using an ETL framework vs. writing yet another ETL script

#68

Post by ideaoverload »

I am tasked with standardizing ETL in a small org, airbyte is on my list to evaluate. As I write this there are 64 comments in the thread and only a single comment mentions actual experience using airbyte. Any other actual insights using the tool? What about Airflow downsides mentioned in the article? Thx!

Re: Using an ETL framework vs. writing yet another ETL script

#69

Post by matesz »

ideaoverload wrote: post #68

I am tasked with standardizing ETL in a small org, airbyte is on my list to evaluate. As I write this there are 64 comments in the thread and only a single comment mentions actual experience using airbyte. Any other actual insights using the tool? What about Airflow downsides mentioned in the article? Thx!

I'm in the same situation!

Re: Using an ETL framework vs. writing yet another ETL script

#70

Post by idiocrat »

ideaoverload wrote: post #68

I am tasked with standardizing ETL in a small org, airbyte is on my list to evaluate. As I write this there are 64 comments in the thread and only a single comment mentions actual experience using airbyte. Any other actual insights using the tool? What about Airflow downsides mentioned in the article? Thx!

Have not worked with the tool myself. Sorry cannot help.

YT has usually good tutorials and concept explanations. Maybe a good point to start.

https://www.youtube.com/results?search_query=airbyte

Post reply on HN