Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Launch HN: Airbyte (YC W20) – Open-Source ELT (Fivetran/Stitch Alternative)

news.ycombinator.com

Post reply on HN
Print view View original

41–50 of 90 posts

Re: Launch HN: Airbyte (YC W20) – Open-Source ELT (Fivetran/Stitch Alternative)

#41

Post by jeanlaf »

noahmbarr wrote: post #13
georgewfraser wrote: post #8

The fundamental challenge of open-source ETL is that high-quality connectors require understanding and working around all kinds of corner cases in the API of each data source. It’s very hard to get open source contributors to do this kind of work; it’s a real slog. Hence at Fivetran we’ve always stuck with the commercial route.

A customer perspective from a mid-stage CFO: I like saving money, but prefer to pay for software solutions like this directly. I pay you, and you make sure this set of connectors {in and out} continues to durably work. Meanwhile, our engineers can focus on building our product.

This is something we will definitely offer as well, with an SLA. And because the maintenance is not only done by us, but the community as well, fixes will be propagated throughout all users much faster than if it has to go through customer support.

Open-source doesn't mean you can't have both. You can check how Databricks or Confluent are doing.

Re: Launch HN: Airbyte (YC W20) – Open-Source ELT (Fivetran/Stitch Alternative)

#42

Post by llampx »

giovannibonetti wrote: post #6

Shouldn't it be ETL (extract, transform, load) [1]? [1] https://en.wikipedia.org/wiki/Extract%2C_transform%2C_load

ETL and ELT are similar but separate things. ELT has been popular for a few years now, and the data lake approach as well as cheap storage has solidified it as the current preferred way to do a data warehouse.

Re: Launch HN: Airbyte (YC W20) – Open-Source ELT (Fivetran/Stitch Alternative)

#43

Post by theboat »

jeanlaf wrote: post #33
influx wrote: post #14

How do you differentiate from Meltano?

The main difference is that Meltano is based on Singer while Airbyte is based on the Airbyte Protocol. We don’t believe Singer is a good building block, because it requires a significant time investment from its users to compensate for the absence of centralized enforcement of the Singer protocol. Since it is not enforced, there is often no guarantee that any pair of Singer connectors are compatible. It defeats the p…

Does this imply airbyte only supports connectors that airbyte validates and integrates into the platform? Can I use an airbyte connector that lives in a repo on my private github?

This solves the problem of getting high quality connectors built, but how do you plan to maintain them? What if the original contributor falls off the face of the earth?

Re: Launch HN: Airbyte (YC W20) – Open-Source ELT (Fivetran/Stitch Alternative)

#44

Post by mtricot »

theboat wrote: post #43
jeanlaf wrote: post #33

Earlier quoted context omitted.

The main difference is that Meltano is based on Singer while Airbyte is based on the Airbyte Protocol. We don’t believe Singer is a good building block, because it requires a significant time investment from its users to compensate for the absence of centralized enforcement of the Singer protocol. Since it is not enforced, there is often no guarantee that any pair of Singer connectors are compatible. It defeats the p…

Does this imply airbyte only supports connectors that airbyte validates and integrates into the platform? Can I use an airbyte connector that lives in a repo on my private github? This solves the problem of getting high quality connectors built, but how do you plan to maintain them? What if the original contributor falls off the face of the earth?

You can also use an external connector if you want to. This is a very valid use-case, especially if you connect internal APIs or private sources that wouldn't make sense for the community.

If the original contributor falls off the face of the earth, it is OK! That's the beauty of Open-Source. Another person who is using it can jump in. We can also jump in.

Re: Launch HN: Airbyte (YC W20) – Open-Source ELT (Fivetran/Stitch Alternative)

#45

Post by polskibus »

Some hard questions that I need to ask as someone that does set up complex data pipelines for customers as part of his job.

How do you want to monetize your product? What is your runway as of today? When do you project you will be self-sustainable?

It's all great to have an open source solution for pushing the data around, but I don't want to invest in learning a new tool only to see it vanish in 2-3 years or so.

Re: Launch HN: Airbyte (YC W20) – Open-Source ELT (Fivetran/Stitch Alternative)

#46

Post by bithavoc »

mtricot wrote: post #37
bithavoc wrote: post #34

Is there a guide to set it up behind a reverse proxy with https? I have it running with Nginx but it won't load, it insist in using /8001/api. How do I change the API endpoint?

You can. You can play the the env variable: API_URL For example if you have a reverse proxy that serves both the webapp and the api you can just launch with: API_URL=/api/v1/ docker-compose up

I just effortlessly synced Freshdesk to Postgres in 20 minutes. Thank you.

Re: Launch HN: Airbyte (YC W20) – Open-Source ELT (Fivetran/Stitch Alternative)

#47

Post by jeanlaf »

polskibus wrote: post #45

Some hard questions that I need to ask as someone that does set up complex data pipelines for customers as part of his job. How do you want to monetize your product? What is your runway as of today? When do you project you will be self-sustainable? It's all great to have an open source solution for pushing the data around, but I don't want to invest in learning a new tool only to see it vanish in 2-3 years or so.

Sure! Those are actually great questions!

Regarding monetization, you can see more details here: https://airbyte.io/pricing We consider 2 monetization approaches: Open core (connectors staying open-source forever) with premium features as: hosting & management (cloud-based control panel without access to your data plan), and enterprise features (privacy compliance, SSO, user access management, etc) What we call “Powered by Airbyte” where we enable you to offer integrations to your own customers using our API

Regarding our runway, with the team as is, mid 2025. We intend to grow the team though, given the adoption growth we have. We’ve already been approached for a Series-A, but will consider it in mid 2022.

Regarding self-sustainability, do you mean financially? Possibly at the end of 2023 or 2024.

How does that sound to you? Genuinely curious.

Re: Launch HN: Airbyte (YC W20) – Open-Source ELT (Fivetran/Stitch Alternative)

#48

Post by polskibus »

jeanlaf wrote: post #47
polskibus wrote: post #45

Some hard questions that I need to ask as someone that does set up complex data pipelines for customers as part of his job. How do you want to monetize your product? What is your runway as of today? When do you project you will be self-sustainable? It's all great to have an open source solution for pushing the data around, but I don't want to invest in learning a new tool only to see it vanish in 2-3 years or so.

Sure! Those are actually great questions! Regarding monetization, you can see more details here: https://airbyte.io/pricing We consider 2 monetization approaches: Open core (connectors staying open-source forever) with premium features as: hosting & management (cloud-based control panel without access to your data plan), and enterprise features (privacy compliance, SSO, user access management, etc) What we call “Powe…

Your pricing page does not have any prices so it's hard to tell. I'll definitely keep an eye on your progress though. Your connector as container approach is interesting although i think IT departments may have issues with an on prem solution that spawns other containers by itself.

One more thing - how will you protect yourself from let's say AWS forking you and selling a managed airbyte version?

Re: Launch HN: Airbyte (YC W20) – Open-Source ELT (Fivetran/Stitch Alternative)

#49

Post by appleflaxen »

polskibus wrote: post #48
jeanlaf wrote: post #47

Earlier quoted context omitted.

Sure! Those are actually great questions! Regarding monetization, you can see more details here: https://airbyte.io/pricing We consider 2 monetization approaches: Open core (connectors staying open-source forever) with premium features as: hosting & management (cloud-based control panel without access to your data plan), and enterprise features (privacy compliance, SSO, user access management, etc) What we call “Powe…

Your pricing page does not have any prices so it's hard to tell. I'll definitely keep an eye on your progress though. Your connector as container approach is interesting although i think IT departments may have issues with an on prem solution that spawns other containers by itself. One more thing - how will you protect yourself from let's say AWS forking you and selling a managed airbyte version?

This is a key question. Licensing via GPL-3 would provide some protection against this, so can you explain why you went with MIT?

Re: Launch HN: Airbyte (YC W20) – Open-Source ELT (Fivetran/Stitch Alternative)

#50

Post by jeanlaf »

polskibus wrote: post #48
jeanlaf wrote: post #47

Earlier quoted context omitted.

Sure! Those are actually great questions! Regarding monetization, you can see more details here: https://airbyte.io/pricing We consider 2 monetization approaches: Open core (connectors staying open-source forever) with premium features as: hosting & management (cloud-based control panel without access to your data plan), and enterprise features (privacy compliance, SSO, user access management, etc) What we call “Powe…

Your pricing page does not have any prices so it's hard to tell. I'll definitely keep an eye on your progress though. Your connector as container approach is interesting although i think IT departments may have issues with an on prem solution that spawns other containers by itself. One more thing - how will you protect yourself from let's say AWS forking you and selling a managed airbyte version?

The pricing depends on the number of connectors you need (but not the volume of data), the features you use, and the SLA. We already saw that there was a great willingness to pay, so we’re not really worried about that part. We will learn a great deal about that next year. Our sole focus in 2021 is the open-source part of the project, and becoming the new standard to replicate data.

Regarding cloud providers as AWS, there are several things we can do to monetize this part with them (including with “Powered by Airbyte”). But, honestly, that would be a great problem, as it would indicate that we’ve become the standard way to replicate data.

Post reply on HN