Launch HN: Airbyte (YC W20) – Open-Source ELT (Fivetran/Stitch Alternative)

Michel here with John, Shrif, Jared, Charles, and Chris. We are building an open-source ELT platform that replicates data from any applications, APIs, databases, etc. into your data warehouses, data lakes or databases: https://airbyte.io.

I’ve been in data engineering for 11 years. Before Airbyte, I was the head of integrations at Liveramp, where we built and scaled over 1,000 data ingestion connectors to replicate 100TB worth of data every day. John, on the other end, has already built 3 startups with 2 exits. His latest one didn’t work out, though. He spent almost a year building ETL pipelines for an engineering management platform, but he eventually ran out of money before reaching product-market fit.

By late 2019, we had known each other for 7 years, and always wanted to work together. When John’s third startup shut down, it was finally the right timing for both of us. And we knew which problem we wanted to address: data integration, and ELT more specifically.

We started interviewing Fivetran, Stitchdata, and Matillion’s customers, in order to see if the existing solutions were solving their problems. We learned they all fell short, and always with the same patterns.

Some limitations we identified are due to the fact that they are closed source. This prevents them from addressing the long tail of integrations because they will always have a ROI consideration when building and maintaining new connectors. A good example is Fivetran which, after 8 years, offers around 150 connectors. This is not a lot when you look at the number of existing tools out there (more than 10,000). In fact, all their customers that we talked to are building and maintaining their own connectors (along with orchestration, scheduling, monitoring, etc.) in-house, as the connectors they needed were either not supported in the way they needed or not supported at all.

Some of those customers also tried to leverage existing open-source solutions, but the quality of the existing connectors is inconsistent, as many haven't been updated in years. Plus, they are not usable out of the box.

That’s when we knew we wanted Airbyte to be open-source (MIT license), usable out of the box, and cover the long tail of integrations. By making it trivial to build new connectors on Airbyte in any language (they run as Docker containers), we hope the community will help us build and maintain the long tail of connectors. While open-source also enables us to address all use cases (including internal DBs and APIs), it also allows us to solve the problem inherent to cloud-based solutions: the security and privacy of your data. Companies don’t need to trust yet another 3rd-party vendor. Because it is self-hosted, it will disrupt the pricing of existing solutions.

Here’s a 2-minute demo video if you want to check out how it looks: https://www.youtube.com/watch?v=sKDviQrOAbU

Airbyte can run on a single node without any external infrastructure. We also integrate with Kubernetes (alpha), and will soon integrate with Airflow so you can run replication tasks across your cluster.

Today, our early version supports about 41 sources and 6 destinations (https://docs.airbyte.io/integrations/destinations). We’re releasing new connectors (https://docs.airbyte.io/changelog/connectors) every week (6 of them have already been contributed by the community). We bootstrapped some connectors using the highest-quality ones from Singer. Our connectors will always remain open-source.

Our goal is to solve data integration for as many companies as possible, and the success of Airbyte is predicated on the open-source project becoming loved and ubiquitous. For this reason, we will focus the entirety of 2021 strengthening the open-source edition; we are dedicated to making it amazing for all users. We will eventually create a paid edition (open core model) with enterprise-level features (support, SLA, hosting and management, privacy compliance, role and access management, SSO, etc.) to address the needs of our most demanding users.

Give it a spin: https://github.com/airbytehq/airbyte/ & https://demo.airbyte.io. Let us know what you think. This is our first time building an open-source technology, so we know we have a lot to learn!

@mtricot -- You mention that a big value prop of Airbyte is providing an interface for building custom connectors. Have there been interesting learnings on designing an ideal "interface" to provide developers? How does the interface you provide compare to that of Fivetran's Functions offering [1]?

[0]: https://hightouch.io

[1]: https://fivetran.com/docs/functions

Amongst the main reasons: Singer seems to have been abandoned by StitchData (after they got acquired by Talend), the quality of the connectors is too unpredictable, Singer connectors are not usable outside the box.

We would have preferred to use an existing standard if one already existed. It was a tough decision for us to create something from scratch but now we are very satisfied with the decision. It is way easier for the community and for us to build connectors that meet quality standards and we can make it MIT so the community can have control on the evolution of the protocol.

We actually wrote a few articles about it:

https://docs.airbyte.io/faq/differences-with.../singer-vs-ai...

https://airbyte.io/articles/data-engineering-thoughts/airbyt...

In the old approach, you would run the transform BEFORE loading data into the warehouse. The disadvantage of that approach is that you loose all fidelity of the raw data.

In the new approach (Airbyte's approach), you load the raw data into the warehouse, and then run your transform jobs in the warehouse. You can do that because modern warehouses are cheap and scalable. The benefit of that approach is that you keep your raw data with all its fidelity, opening up endless opportunities for exploratory slicing and dicing.

That's why it's called "ELT" (new) these days, to distinguish from "ELT" (old).