Using an ETL framework vs. writing yet another ETL script

We currently are using Airflow for ELTs/ETLs to ingest from different Postgres databases to BigQuery/Google Cloud Storage. Airbyte looks sweet for the same task and would free us from a big effort burden, but its Postgres source only supports SELECT * statements (i.e. you can't deselect columns).

That's kind of a dealbreaker for us, because for security reasons our Postgres users permissions are granularly configured with column-based security. I hope the Airbyte team solves this eventually because the software is looking great.

That's crazy, why would it have the approach of "select all columns" when you might never need all of them? Could you create a view for it to select from instead?

Either way if someone told me a ETL supports only literally all the columns or nothing then I'd assume it was a MVP product to demonstrate the idea... not a production system.

Airbyte Engineer here.

I think some of the points made here about ETL scripts being just 'ETL scripts' are very relevant. Definitely been on the other side of the table arguing for a quick 3-hour script.

Having written plenty of ETL scripts - in Java with Hadoop/Spark, Python with Airflow and pure Bash - that later morphed into tech debt monsters, I think many people underestimate how quickly these can quickly snowball into proper products with actual requirements.

Unless one is extremely confident an ETL script will remain a non-critical good-to-have part of the stack, I believe evaluating and adopting a good ETL framework, especially one with pre-built integrations is good case of 'sharpening the axe before cutting the tree' and well worth the time.

We've been very careful to minimise Airbyte's learning curve. Starting up Airbyte is as easy as checking out the git repo and running 'docker compose up'. A UI allows users to select, configure and schedule jobs from a list of 120+ supported connectors. It's not uncommon to see users successfully using Airbyte within tens of mins.

If a connector is not supported, we offer a Python CDK that lets anyone develop their own connectors in a matter of hours. We have a commitment to supporting community contributed connectors so there is no worry about contributions going to waste.

Everything is open source, so anyone is free to deep as dive as they need or want to.

We also build in the open and have single-digit hour Slack response time on weekdays. Do check us out - https://github.com/airbytehq/airbyte!

I just got out of the job where we were working on legacy ETL "script" in Elixir, and terrible code architecture decisions aside, I think the pattern where you have to launch and monitor long lasting jobs is a breeze in BEAM. You just spawn one process for each job, report back via mailbox and monitor via Supervisor. Unfortunately making changes to that system where all sources were hardcoded was to say at least abysmal, but the job running core was quite elegant.

Hopefully Elixir and other BEAM compiled languages will gain enough traction, I can't imagine rewriting something that available in Erlang from the box in OOP languages with mutable objects.

At our company, we actually built ETL-Framework-agnostic wrappers, monitoring, logging and scheduling tooling around the different ETL tools we used for four different ETL Product Frameworks we used: Microfocus COBOL, Torrent Orchestrate, Datastage (which incorporated Torrent) and Abinitio. The wrappers invoked the ETL command, reformatted and consolidated logs. For scheduling, we relied mostly on CA Autosys, instead of whatever scheduling mechanisms came with the ETL Product.

We found this approach made it easier to transition from one product to another. As it consistently faster to plug the ETL framework into the supporting framework than to implement everything a new ETL Product offered.

As we move from our on-prem environment to the cloud, we hope we can implement a similar strategy even if we have to switch the support frameworks.