Airbyte: Simple and extensible open-source EL(T)
1–10 of 25 posts
Re: Airbyte: Simple and extensible open-source EL(T)
#2Is this meant as a free version of say DataFactory/FiveTran? I’m always interested in how these products do the change tracking and incremental updates. FiveTran is seamless, DF, more manual configuration. I see they don’t even support incremental updates for MSSQL/PostgreSQL so I’m wondering who would be using this tool?
Re: Airbyte: Simple and extensible open-source EL(T)
#3Is this meant as a free version of say DataFactory/FiveTran? I’m always interested in how these products do the change tracking and incremental updates. FiveTran is seamless, DF, more manual configuration. I see they don’t even support incremental updates for MSSQL/PostgreSQL so I’m wondering who would be using this tool?
Sounds kind of strange not having incremental replication, it would make no sense for an ETL tool. In fact the documentation for PostgreSQL [1] is somehow contradictory at it both states "Incremental sync (copying only the data that has changed) for this source is coming soon" but also "Feature: Incremental - Append Sync / Supported: Yes"
Re: Airbyte: Simple and extensible open-source EL(T)
#4I've found small scripts do a great job with ETL. Run by cron and log to text seems to do the job.
What's the advantages in using a large tool like this?
Re: Airbyte: Simple and extensible open-source EL(T)
#5I saw this project a few months ago and I'm really surprised how fast they've been adding new connectors [1]. I was initially a little doubtful about them implementing their own protocol [2] instead of using Singer [3], already OOS, but it looks like it panned out for them. In fact I've been using Stitch Data for a couple of years and I've find myself "fighting" with their support team about bugs I think we're Singer related, mainly with DDL commands.
[1] https://docs.airbyte.io/changelog/connectors [2] https://airbyte.io/articles/data-engineering-thoughts/why-yo... [3] https://www.singer.io/
Re: Airbyte: Simple and extensible open-source EL(T)
#6I've found small scripts do a great job with ETL. Run by cron and log to text seems to do the job. What's the advantages in using a large tool like this?
It's intended for syncing data sources to a data warehouse for analysis. Even if you're only doing one data source, if you want analysis to be easy later you'll likely want to sync all fields available from the api (as you probably don't know what fields are of interest before analysis).
If you imagine doing this for Stripe say, there's a huge amount of fields available in different objects (charges, invoices, subscriptions etc) and you need to add these as columns to your relation database of a data warehouse. Unnesting may also be needed. That's very tedious work and on top of it you need to run, monitor and maintain the extraction process.
Even a small company could easily have 10+ data sources containing dozens of tables that they want to sync to a warehouse so this quickly becomes unmanageable. Hence companies like Stitchdata, Fivetran and now Airbyte now selling it as a service.
Re: Airbyte: Simple and extensible open-source EL(T)
#7I've found small scripts do a great job with ETL. Run by cron and log to text seems to do the job. What's the advantages in using a large tool like this?
It's a bit unclear on what the scope of this tool is.
But there are a couple of new classes of tools for ETL/ELT or data engineering as it's called now.
There's the "Data Integration Tools" like Fivetran, Stitch, and this. They are collections of connectors that they have coded to ease ingesting data from lots of different database products and stores to another. That's valuable, I wouldn't start writing my own script to pull changes from my RDBMS' WAL to the data warehouse because it's complicated, and if someone can pull those JSON files from whatever cloud storage for me it's good because it's too simple to waste time on.
Then there's DBT and Dataform, those are orchestration tools and development environments all in one. When you start splitting your scripts (SQL queries in this case) in stages either for ease of understanding or efficiency, you'll want to see the dependencies laid out and having them execute in order. They also provide (git) version control, so it makes it a breeze for engineers to manage the pipeline like they would with other software assets, and you can get people from other backgrounds contributing in a more engineering-like workflow, talking about analysts who devise business dashboards and such.
Re: Airbyte: Simple and extensible open-source EL(T)
#8I've found small scripts do a great job with ETL. Run by cron and log to text seems to do the job. What's the advantages in using a large tool like this?
Let's say you're loading data from Postgres into Snowflake. Pretty simple integration ETL. You need to:
* Make sure you're only loading incremental updates
* Make sure you don't miss any data while doing incremental updates. This includes deleted rows.
* Update the Snowflake table schemas as Postgres schemas change. If this is impossible you need to alert someone.
* Keep historical metrics so you know if the job is slow or too much data or whatever.
Now do this for twenty different data sources every hour.
Re: Airbyte: Simple and extensible open-source EL(T)
#9What's EL?
Re: Airbyte: Simple and extensible open-source EL(T)
#10What's EL?
Extract and load. It’s a variation of ETL where you push the Transformation(T) step into the warehouse as opposed to doing the T out of band in code.
This allows you to retain the data as raw as possible and empowers your analytics teams to move faster by not having to wait for engineers to transform the data into a usable form.