Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Airbyte: Simple and extensible open-source EL(T)

github.com

Post reply on HN
Print view View original

11–20 of 25 posts

Re: Airbyte: Simple and extensible open-source EL(T)

#11

Post by glogla »

sdepablos wrote: post #5

I saw this project a few months ago and I'm really surprised how fast they've been adding new connectors [1]. I was initially a little doubtful about them implementing their own protocol [2] instead of using Singer [3], already OOS, but it looks like it panned out for them. In fact I've been using Stitch Data for a couple of years and I've find myself "fighting" with their support team about bugs I think we're Singer…

If you don't mind me asking, how does Stitch scale?

We're currently using bespoke Spark-based ETL tool because we need to move like 2 TB/day from various place, especially legacy system that don't make incremental data movement practical.

How does Stitch even run, all SaaS in Stich Cloud or something?

Thanks!

Re: Airbyte: Simple and extensible open-source EL(T)

#12

Post by sherifnada »

laichzeit0 wrote: post #2

Is this meant as a free version of say DataFactory/FiveTran? I’m always interested in how these products do the change tracking and incremental updates. FiveTran is seamless, DF, more manual configuration. I see they don’t even support incremental updates for MSSQL/PostgreSQL so I’m wondering who would be using this tool?

Hi, Airybte engineer here! We do support incremental updates for MSSQL & Postgres -- I updated the docs to make this clear.

More broadly as an ELT tool, Airbyte is focused on supporting the long tail of integrations. The number of data silos is increasing rapidly, and it's intractable for a single engineering team or company to integrate with them all. So as an OSS tool, Airbyte provides a core platform for replicating data (e.g: a UI, scheduling, logging, state management, etc..) and the most popular connectors out of the box, then gets out of the way of its users. If you need a connector you need that Airbyte doesn't already provide, you can implement a Docker image which conforms to the Airbyte protocol, then start using it in your Airbyte instance without ever needing to go through us (Although many users do end up contributing their connectors back to the core codebase).

Re: Airbyte: Simple and extensible open-source EL(T)

#13

Post by dal »

Just to clarify, this is not a fully open source project. More like freemium or just enough for the community to contribute and maintian free connectors which make it usefull. If you want security, you have to pay. There are non-free parts to it that you are not allowed to use. And contributing these features would probably not be allowed.

Re: Airbyte: Simple and extensible open-source EL(T)

#14

Post by specialist »

xupybd wrote: post #4

I've found small scripts do a great job with ETL. Run by cron and log to text seems to do the job. What's the advantages in using a large tool like this?

Now imagine those self contained scripts are deployed to target servers. Add logging, metrics, remote control, atomic data pipelines.

Or what I like to call 'data processing'.

That's more or less what I built for healthcare IT. Before the term "serverless" was coined.

Just peeked at Airbyte. I'd definitely look longer next time I have to do ETL.

--

My only pause for concern is their phrase "optional normalized schemas". What does that mean?

In my experience, data processing is best treated like screen or web scrapping. The tools based on schemas, mappings, patch cords (visual programing) always prove useless. All the CASE and workflowy tools like BizTalk and Talend are 800lb angry gorillas siting between you and your work.

Super simple self contained scripts, which can easily be ran from the command line or REPL, are The Correct Answer™.

Re: Airbyte: Simple and extensible open-source EL(T)

#15

Post by sherifnada »

dal wrote: post #13

Just to clarify, this is not a fully open source project. More like freemium or just enough for the community to contribute and maintian free connectors which make it usefull. If you want security, you have to pay. There are non-free parts to it that you are not allowed to use. And contributing these features would probably not be allowed.

Hi, Airbyte engineer here.

All connectors and the core functionality of Airbyte will _always_ be open source and free to use. You will always be able to download and operate an instance of Airbyte, containing all connectors and ELT features (whether for personal projects or business operations in your company) for free.

However, there are some things that we might offer as a monetized product in the future, such as:

* enterprise compliance requirements (E.g SOC2)

* SSO

* Hosting Management

Hope this clarifies the direction we’re taking.

Re: Airbyte: Simple and extensible open-source EL(T)

#16

Post by cgardens »

specialist wrote: post #14
xupybd wrote: post #4

I've found small scripts do a great job with ETL. Run by cron and log to text seems to do the job. What's the advantages in using a large tool like this?

Now imagine those self contained scripts are deployed to target servers. Add logging, metrics, remote control, atomic data pipelines. Or what I like to call 'data processing'. That's more or less what I built for healthcare IT. Before the term "serverless" was coined. Just peeked at Airbyte. I'd definitely look longer next time I have to do ETL. -- My only pause for concern is their phrase "optional normalized schema…

Different Airbyte engineer here!

Wanted to help answer your question as to what "optional normalized schemas" means. When writing data into your data warehouse we provide 2 options: 1. write each record as a json blob. 2. infer the schema of the data and write each value in a record to its own column with an appropriate type.

We are betting on EL(T), meaning we think that Transform should be considered separately from EL. To give a more a concrete example, if you are already using DBT in your data warehouse to normalize your data, you likely prefer operating on the "raw" (json blob) data than an arbitrarily normalized form of your data that your EL pipeline has decided on for you. I am seeing this trend pretty pervasively where a lot of, nominally, ELT pipelines are outsourcing the Transform to best in breed tools like DBT.

Thanks for asking this question btw, we'll do our best to clarify in our docs!

Re: Airbyte: Simple and extensible open-source EL(T)

#17

Post by cgardens »

xupybd wrote: post #4

I've found small scripts do a great job with ETL. Run by cron and log to text seems to do the job. What's the advantages in using a large tool like this?

(Airbyte Engineer)

I think what you're saying here is often true until it isn't. For a personal project where you're pulling data from one API? Sure.

Once you have an engineering system with multiple engineers relying on the data to be pulled reliably, having a host of individual ELT crons gets brittle really fast. At the past couple companies I've worked out this same narrative has played out:

"Oh we need to pull data from X let's build a cron." (3 months later.) "Wait a second why is all of this data 1 month old? Oh the cron hasn't run in a month because of a schema change. Let's add monitoring." (3 months later.) "We need to change the cron to pull fields A,B,C hourly and field D,E,F weekly." (3 months later.) "The amount of data we're pulling is making this too expensive, we need to implement some sort of incremental replication." ... etc

It always starts out as a "small" script but they rarely stay that way. In my experience, they end up needing the same features that get rebuilt over and over again on an ad hoc basis. We want an engineer to be able to get these features out of the box.

We generally think that for most engineering teams (even pretty small ones) the ad hoc crons for pulling data become nightmarish pretty fast. This problem is compounded if you are already using some other ETL as a service provider but they don't support one of your data sources so you also have a separate set of crons. By taking an OSS approach we're trying to cover that long tail, so that all of your ELT can be managed using one tool.

Re: Airbyte: Simple and extensible open-source EL(T)

#18

Post by jeanlaf »

sdepablos wrote: post #3
laichzeit0 wrote: post #2

Is this meant as a free version of say DataFactory/FiveTran? I’m always interested in how these products do the change tracking and incremental updates. FiveTran is seamless, DF, more manual configuration. I see they don’t even support incremental updates for MSSQL/PostgreSQL so I’m wondering who would be using this tool?

Sounds kind of strange not having incremental replication, it would make no sense for an ETL tool. In fact the documentation for PostgreSQL [1] is somehow contradictory at it both states "Incremental sync (copying only the data that has changed) for this source is coming soon" but also "Feature: Incremental - Append Sync / Supported: Yes" [1] https://docs.airbyte.io/integrations/sources/postgres

(Airbyte engineer) Agreed. Thanks for notifying us about that. We fixed it. We started by supporting only full refresh at the very start, as it was easier to support more connectors, but have been ramping up incremental across all the sources for a month now. Incremental should be supported for all sources within a few weeks.

Re: Airbyte: Simple and extensible open-source EL(T)

#19

Post by sdepablos »

glogla wrote: post #11
sdepablos wrote: post #5

I saw this project a few months ago and I'm really surprised how fast they've been adding new connectors [1]. I was initially a little doubtful about them implementing their own protocol [2] instead of using Singer [3], already OOS, but it looks like it panned out for them. In fact I've been using Stitch Data for a couple of years and I've find myself "fighting" with their support team about bugs I think we're Singer…

If you don't mind me asking, how does Stitch scale? We're currently using bespoke Spark-based ETL tool because we need to move like 2 TB/day from various place, especially legacy system that don't make incremental data movement practical. How does Stitch even run, all SaaS in Stich Cloud or something? Thanks!

Pure SaaS. Check their pricing [1], as is row based and depending on how much data you need to move it can easily go out of budget ;)

[1] https://www.stitchdata.com/pricing/

Re: Airbyte: Simple and extensible open-source EL(T)

#20

Post by bleonard »

forgetfulness wrote: post #7
xupybd wrote: post #4

I've found small scripts do a great job with ETL. Run by cron and log to text seems to do the job. What's the advantages in using a large tool like this?

It's a bit unclear on what the scope of this tool is. But there are a couple of new classes of tools for ETL/ELT or data engineering as it's called now. There's the "Data Integration Tools" like Fivetran, Stitch, and this. They are collections of connectors that they have coded to ease ingesting data from lots of different database products and stores to another. That's valuable, I wouldn't start writing my own scrip…

Just for fun and completeness, there's also taking action on this data outside of your warehouse by making your external tools smarter - in some cases, even writing _back_ to the things you EL'd into it. For example, writing product usage to Salesforce. These are the kinds of things we are focusing on at Grouparoo, also an open source project.
Post reply on HN