Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Launch HN: Airbyte (YC W20) – Open-Source ELT (Fivetran/Stitch Alternative)

news.ycombinator.com

Post reply on HN
Print view View original

21–30 of 90 posts

Re: Launch HN: Airbyte (YC W20) – Open-Source ELT (Fivetran/Stitch Alternative)

#21

Post by cgardens »

georgewfraser wrote: post #8

The fundamental challenge of open-source ETL is that high-quality connectors require understanding and working around all kinds of corner cases in the API of each data source. It’s very hard to get open source contributors to do this kind of work; it’s a real slog. Hence at Fivetran we’ve always stuck with the commercial route.

We couldn’t agree more that producing high-quality connectors requires a lot of work. The hardest part about this task is that connectors must evolve quickly (due to changes in the API, new corner cases, etc). The quality of the connector is not just how well the first version works but how well it works throughout its entire lifetime.

Our perspective is that by providing these connectors as open source we can arrive at higher quality connectors. For a closed source solution, a user has to go through customer service and persuade them that there is indeed a problem. A story we have heard countless times, is that SaaS ETL providers are slow to fix corner cases discovered by users leading to extended downtime. With an OSS solution, a user can fix a problem themselves and be back online immediately.

We proactively maintain all connectors, but we believe that by sharing that responsibility with the OSS community, we can achieve the highest quality connectors.

One of the main focuses of Airbyte is to provide a very strong open-source MIT standard for testing and developing (base packages, standard tests, best practices…) connectors in order to achieve the highest quality.

Re: Launch HN: Airbyte (YC W20) – Open-Source ELT (Fivetran/Stitch Alternative)

#22

Post by vianneychevalie »

NavyDish wrote: post #12
georgewfraser wrote: post #8

The fundamental challenge of open-source ETL is that high-quality connectors require understanding and working around all kinds of corner cases in the API of each data source. It’s very hard to get open source contributors to do this kind of work; it’s a real slog. Hence at Fivetran we’ve always stuck with the commercial route.

Similar thoughts (btw I came here looking for your comment ha!). I guess you had mentioned in one of the videos that at Fivetran, it is your responsibility to ensure data integrity across all of the sources/integrations, and has been since the early days. This led the customers to trust the product in the early days and the team to draw learnings from abstract patterns across sources. Have come to believe that it is…

How do you determine this explicit ownership for issues? I've come across many governance problems linked to a lack of transparency in "bug ownership", but I've often failed to find a common ground for clients and third parties: who's responsible? Who should pay for it?

Quite often it's the one with the loudest mouth or the biggest sponsor who wins.

Re: Launch HN: Airbyte (YC W20) – Open-Source ELT (Fivetran/Stitch Alternative)

#24

Post by mtricot »

satyrnein wrote: post #16

I notice dbt integration on your roadmap. As a current Stitch and dbt Cloud customer, can you give some insight into what you're considering there?

Yes, we love DBT too!

It’s great for handling transformations and since we want to focus on the EL part, we think there’s good synergy there.

Airbyte is already using the DBT CLI internally and as we provide more transformations of the data during syncs, we’ll make it easier to give a better integration with DBT projects downstream:

- Native Transformations as part of sync process: for example, schema migrations for source data changes, un-nesting complex objects columns (from APIs), etc

- Customizable models to override or extend further Airbyte’s proposed transformations to be executed in the same sync pipeline

- Seamless DX between custom downstream transformation and transformations made by Airbyte

- Integration with external orchestrators (Airflow, DBT Cloud jobs) with webhook triggers?

We’re happy to hear more ideas/needs to build this roadmap though!

You can have a look at the current state of Airbyte with DBT here: https://docs.airbyte.io/tutorials/connecting-el-with-t-using...

Re: Launch HN: Airbyte (YC W20) – Open-Source ELT (Fivetran/Stitch Alternative)

#26

Post by rilut »

mtricot wrote: post #25
rilut wrote: post #23

Will it support change data capture (CDC) from SQL databases?

Yes, it is on our roadmap. We will likely have an alpha version in the next two months. https://github.com/airbytehq/airbyte/issues/957

Fantastic. We've using both managed service and in-house solution (debezium + kafka) for CDC from transactional to analytics. Looking forward to see this on Airbyte.

Re: Launch HN: Airbyte (YC W20) – Open-Source ELT (Fivetran/Stitch Alternative)

#27

Post by Dnguyen »

I do a lot of ETL over the years and watched the video and read the intro. I apologize but I haven't a chance to read through the full documentation. My question is, a lot of data I pull are relational and hierarchical, how would I pass variables along to related connectors? And is there a way to wait for a parent connection to finish before the child connector runs? I've build many ETL over the years and it's not easy to keep up with the changes. The bottleneck was always the engineering adapting to the changing schema.

Re: Launch HN: Airbyte (YC W20) – Open-Source ELT (Fivetran/Stitch Alternative)

#29

Post by mtricot »

Dnguyen wrote: post #27

I do a lot of ETL over the years and watched the video and read the intro. I apologize but I haven't a chance to read through the full documentation. My question is, a lot of data I pull are relational and hierarchical, how would I pass variables along to related connectors? And is there a way to wait for a parent connection to finish before the child connector runs? I've build many ETL over the years and it's not ea…

It is not possible as of now but we are about to start integrating with DAG managers (Airflow, Dagster, Prefect...). That will give the possibility to schedule the connectors with the proper dependencies.

See: https://github.com/airbytehq/airbyte/issues/836

Re: Launch HN: Airbyte (YC W20) – Open-Source ELT (Fivetran/Stitch Alternative)

#30

Post by renewiltord »

georgewfraser wrote: post #8

The fundamental challenge of open-source ETL is that high-quality connectors require understanding and working around all kinds of corner cases in the API of each data source. It’s very hard to get open source contributors to do this kind of work; it’s a real slog. Hence at Fivetran we’ve always stuck with the commercial route.

Personally, the most infuriating thing about a tool is where I can fix the damn thing given the source code but I have to go through the support staff to the engineering team and then wait for "this is on our roadmap but not something we're currently prioritizing". Right, I know. I don't expect other people to do work for me. I just need them to let me do the work myself.

The massive advantage of the OSS route isn't that you can ask the community to build a tool for you; it's that when you inevitably have a corner case or some behaviour you want to encode, you can just make RenesPostgres connector and copy in the Postgres connector and fix it.

I don't understand why anyone keeps their source all closed. Even one of those "you can't release this but you can edit it" licenses is better.

Half of why I use Kong as an API Gateway is that I can just edit the source code of their plugins. Thank fuck for that.

Post reply on HN