Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Launch HN: Airbyte (YC W20) – Open-Source ELT (Fivetran/Stitch Alternative)

news.ycombinator.com

Post reply on HN
Print view View original

11–20 of 90 posts

Re: Launch HN: Airbyte (YC W20) – Open-Source ELT (Fivetran/Stitch Alternative)

#11

Post by mtricot »

tejasmanohar wrote: post #3

Congrats! We at Hightouch [0] ("reverse ETL") are excited to see Airbyte here on HN. We've been following Michel & John for a while now since the YC days, and from the outside, it seems like they've been consistently shipping incredibly quickly ever sinec the open-source project launch. @mtricot -- You mention that a big value prop of Airbyte is providing an interface for building custom connectors. Have there been i…

Answering your first question: When talking about the interface we need to separate: the data protocol and the developer experience (DX) creating & maintaining a connector. We believe the data protocol we have in place should address 95% of the use cases and, as we get more sophisticated use cases we will evolve the protocol (for example for more scale). Regarding the DX, we are continuously working on it to make it a breeze and ensure super high quality.

Answering your second question: Fivetran functions are a nice escape hatch but none of the users we talked to mentioned those. They always mentioned building inhouse for missing connectors. My interpretation is that this is too much of a vendor lock-in for a cloud-based product.

Re: Launch HN: Airbyte (YC W20) – Open-Source ELT (Fivetran/Stitch Alternative)

#12

Post by NavyDish »

georgewfraser wrote: post #8

The fundamental challenge of open-source ETL is that high-quality connectors require understanding and working around all kinds of corner cases in the API of each data source. It’s very hard to get open source contributors to do this kind of work; it’s a real slog. Hence at Fivetran we’ve always stuck with the commercial route.

Similar thoughts (btw I came here looking for your comment ha!).

I guess you had mentioned in one of the videos that at Fivetran, it is your responsibility to ensure data integrity across all of the sources/integrations, and has been since the early days. This led the customers to trust the product in the early days and the team to draw learnings from abstract patterns across sources.

Have come to believe that it is THE MOST important thing to have an explicit ownership for issues whenever there is physical movement of data across an org's ecosystem.

Re: Launch HN: Airbyte (YC W20) – Open-Source ELT (Fivetran/Stitch Alternative)

#13

Post by noahmbarr »

georgewfraser wrote: post #8

The fundamental challenge of open-source ETL is that high-quality connectors require understanding and working around all kinds of corner cases in the API of each data source. It’s very hard to get open source contributors to do this kind of work; it’s a real slog. Hence at Fivetran we’ve always stuck with the commercial route.

A customer perspective from a mid-stage CFO: I like saving money, but prefer to pay for software solutions like this directly. I pay you, and you make sure this set of connectors {in and out} continues to durably work. Meanwhile, our engineers can focus on building our product.

Re: Launch HN: Airbyte (YC W20) – Open-Source ELT (Fivetran/Stitch Alternative)

#15

Post by marcinzm »

georgewfraser wrote: post #8

The fundamental challenge of open-source ETL is that high-quality connectors require understanding and working around all kinds of corner cases in the API of each data source. It’s very hard to get open source contributors to do this kind of work; it’s a real slog. Hence at Fivetran we’ve always stuck with the commercial route.

Yeah, Fivetran seems often to have issues that come down to "we had a discussion with data source/sink provider and found they had a bug in their latest release." Even if an open source contributor gets to that point they won't have the strong arm ability to force the provider to fix the bug ASAP.

Re: Launch HN: Airbyte (YC W20) – Open-Source ELT (Fivetran/Stitch Alternative)

#17

Post by DouweM »

jameslk wrote: post #2

It's nice to see some competition in this space, especially open source. I'm a little confused though I thought Singer was the open source version of Stitch (which you mention briefly): https://www.singer.io . But maybe Singer doesn't have all the UI features that Stitch does and that's where Airbyte is different? I would love to know more about the differences

At GitLab, we're not ready to give up on the Singer spec, community, and ecosystem yet, which is why I've been working on Meltano for the past year: https://meltano.com/

We think that the biggest things holding back Singer are the lack of documentation and tooling around taking existing taps and targets to production, and around building, debugging, maintaining, and testing new or existing high-quality taps and targets.

Meltano itself addresses the first problem, and provides a robust and reliable platform for building, running & orchestrating Singer- and dbt-based ELT pipelines. It's built for developers who are comfortable with CLIs and YML files, and want their pipelines to be defined in a Git repository so that they get the benefits of DevOps best practices like code review and CI/CD.

At the same time, we have been working with some members of the community on a new framework for building taps and targets: https://gitlab.com/meltano/meltano/-/issues/2401, which we have decided to call the Singer SDK: https://gitlab.com/meltano/singer-sdk. We are moving as many Singer specification-specific details around things like incremental state replication and stream/field selection into the framework, so that individual taps only need to worry about getting the data from the source and can be expected to behave more consistently and correctly across the board.

Re: Launch HN: Airbyte (YC W20) – Open-Source ELT (Fivetran/Stitch Alternative)

#18

Post by cpard »

georgewfraser wrote: post #8

The fundamental challenge of open-source ETL is that high-quality connectors require understanding and working around all kinds of corner cases in the API of each data source. It’s very hard to get open source contributors to do this kind of work; it’s a real slog. Hence at Fivetran we’ve always stuck with the commercial route.

That's an excellent point and not easy to demonstrate until someone does experience an edge case with their connectors. The main value of open sourcing a framework for integrations (e.g. Singer), is to allow customers to easily support a large number of long tail integrations that exist out there.

Re: Launch HN: Airbyte (YC W20) – Open-Source ELT (Fivetran/Stitch Alternative)

#19

Post by DouweM »

georgewfraser wrote: post #8

The fundamental challenge of open-source ETL is that high-quality connectors require understanding and working around all kinds of corner cases in the API of each data source. It’s very hard to get open source contributors to do this kind of work; it’s a real slog. Hence at Fivetran we’ve always stuck with the commercial route.

I think that if the wider open source community can maintain API client libraries for every imaginable SaaS API and every popular programming language, there's no reason that it can't maintain open source ELT connectors for all of these sources as well.

I work at GitLab as project lead of Meltano (https://meltano.com/) — which embraces Singer instead of abandoning it — and we've seen a lot of interest from data consultancies looking for mature tooling around deploying and developing Singer taps, many of whom have expressed that they'd be happy to maintain open source ELT connectors for data sources that are commonly used by their clients, if they can significantly save on ELT costs that would otherwise get passed on to those clients.

Of course, only one data consultancy (or data team at a company) would need to maintain an open source tap, and others that need the same source for _their_ clients can contribute and help keep it up to date.

Post reply on HN