Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Launch HN: Airbyte (YC W20) – Open-Source ELT (Fivetran/Stitch Alternative)

news.ycombinator.com

Post reply on HN
Print view View original

81–90 of 90 posts

Re: Launch HN: Airbyte (YC W20) – Open-Source ELT (Fivetran/Stitch Alternative)

#81

Post by mtricot »

crowquill wrote: post #80

Congratulations on the launch. I’m just getting started in the data pipelines world and was wondering, how does Airbyte compare to Embulk?

Airbyte addresses two pains of data engineers:

- having to build and maintain connectors

- managing data integration pipelines on behalf of less technical profiles

Because Airtybe is UI/API based, we can offer a great experience to less technical profiles so that they can become independent and leverage data in the most efficient way.

Embulk addresses a different usecase.

Re: Launch HN: Airbyte (YC W20) – Open-Source ELT (Fivetran/Stitch Alternative)

#82

Post by mtricot »

francoisp wrote: post #73

Very nice! Data wants to be free. Please check out this rosettable thing we open-sourced last year, it could come in handy: it allows real time ETL-like sync between mysql and postgres. https://github.com/francoisp/rosettable I've implemented the struts of a framework built around rosettable, with mautic davical as a proof of concept, it keeps a user defined map (one to one, many to one, one to many and many to many)…

Thanks! will look at it!

Re: Launch HN: Airbyte (YC W20) – Open-Source ELT (Fivetran/Stitch Alternative)

#83

Post by kiwicopple »

yclurker wrote: post #78
kiwicopple wrote: post #62

Earlier quoted context omitted.

hey yclurker, Supabase founder here. I'm sorry we haven't delivered a better self-hosting experience. This is clearly something we could do better. As I mentioned in your link[1], we're targeting a release of our CLI in Q1 (last week of march). > its a been long time since their launch It has been 8 months since our alpha launch*, and just over 1 month since our beta. I hope that sets some context, because personally…

@kiwicopple : I'm really sorry to say this, I understand you try your best to sound reasonable every time. It's just that you alone believe that we can't see past your replies. But it is pretty evident that you wont make Supabase self hostable. Dude, even proprietary closed products provide docker / docker-compose by default in their repos. Quite honestly, shame on Supabase for setting such a shady example.

In the link you shared I mention how to get the docker-compose file by running `supabase eject`

I probably can't satisfy you, but I'll make our intentions very clear to everyone else reading this:

https://github.com/supabase/supabase/commit/913add2e3ca45e55...

Sorry for the lack of documentation - we will add more docs and make everything easier to use before a "Launch week" that we have planned on the last week of March.

Re: Launch HN: Airbyte (YC W20) – Open-Source ELT (Fivetran/Stitch Alternative)

#84

Post by yclurker »

kiwicopple wrote: post #83
yclurker wrote: post #78

Earlier quoted context omitted.

@kiwicopple : I'm really sorry to say this, I understand you try your best to sound reasonable every time. It's just that you alone believe that we can't see past your replies. But it is pretty evident that you wont make Supabase self hostable. Dude, even proprietary closed products provide docker / docker-compose by default in their repos. Quite honestly, shame on Supabase for setting such a shady example.

In the link you shared I mention how to get the docker-compose file by running `supabase eject` I probably can't satisfy you, but I'll make our intentions very clear to everyone else reading this: https://github.com/supabase/supabase/commit/913add2e3ca45e55... Sorry for the lack of documentation - we will add more docs and make everything easier to use before a "Launch week" that we have planned on the last week of M…

From your commit in README.md

>> You can emulate Supabase using `docker-compose` by following these steps inside the `./docker` folder.

What does even emulating mean for an open source product ? Why do I want to emulate and not run the real thing.

>> I probably can't satisfy you

Dude, if you can show me one another product which claims to be "open source" that can't be self hosted I will admit it.

Re: Launch HN: Airbyte (YC W20) – Open-Source ELT (Fivetran/Stitch Alternative)

#85

Post by b-luu »

manishsharan wrote: post #74

qq: how do I sell ELT to my managers who have their heart set on ETL ?

As mentioned by cgardens in reply to another question, you can read through John's (one of the founders of Airbyte) article on the future of ETL/ELT: https://airbyte.io/articles/data-engineering-thoughts/why-th...

It provides some solid arguments that should help your discussion with your managers.

Re: Launch HN: Airbyte (YC W20) – Open-Source ELT (Fivetran/Stitch Alternative)

#86

Post by crunchtoinfo »

yclurker wrote: post #32

The Open Source Fivetran alternative. Yay, it was about time! A simple license : MIT. Clear differentiation between free & paid plans. I am liking what I am seeing so far. One of our client is in advertising industry and is syncing data from 20 different API vendors to postgres. So I am one of your potential customer. However, there is a big problem I'm noticing with "Open source alternatives" lately on HN. I had to…

I can confirm that self-hosted is the best of both worlds: fully private and you control the access through your own infrastructure and all works in the browser. This simplicity is important for many small-medium companies. That is why my go-to tools for adhoc data slicing by non-technical users is https://www.metabase.com. I hope that Airbyte/dbt will be the ELT complement. And I hope to find a way to contribute.

Re: Launch HN: Airbyte (YC W20) – Open-Source ELT (Fivetran/Stitch Alternative)

#87

Post by bleonard »

crunchtoinfo wrote: post #86
yclurker wrote: post #32

The Open Source Fivetran alternative. Yay, it was about time! A simple license : MIT. Clear differentiation between free & paid plans. I am liking what I am seeing so far. One of our client is in advertising industry and is syncing data from 20 different API vendors to postgres. So I am one of your potential customer. However, there is a big problem I'm noticing with "Open source alternatives" lately on HN. I had to…

I can confirm that self-hosted is the best of both worlds: fully private and you control the access through your own infrastructure and all works in the browser. This simplicity is important for many small-medium companies. That is why my go-to tools for adhoc data slicing by non-technical users is https://www.metabase.com . I hope that Airbyte/dbt will be the ELT complement. And I hope to find a way to contribute.

To continue to compliment this stack, feel free to check out Grouparoo, a "reverse ETL" tool that is also self-hosted. https://www.grouparoo.com

Re: Launch HN: Airbyte (YC W20) – Open-Source ELT (Fivetran/Stitch Alternative)

#88

Post by mtricot »

crunchtoinfo wrote: post #86
yclurker wrote: post #32

The Open Source Fivetran alternative. Yay, it was about time! A simple license : MIT. Clear differentiation between free & paid plans. I am liking what I am seeing so far. One of our client is in advertising industry and is syncing data from 20 different API vendors to postgres. So I am one of your potential customer. However, there is a big problem I'm noticing with "Open source alternatives" lately on HN. I had to…

I can confirm that self-hosted is the best of both worlds: fully private and you control the access through your own infrastructure and all works in the browser. This simplicity is important for many small-medium companies. That is why my go-to tools for adhoc data slicing by non-technical users is https://www.metabase.com . I hope that Airbyte/dbt will be the ELT complement. And I hope to find a way to contribute.

Perfectly timed comment :) we are currently writing a tutorial at the moment to showcase how to use Airbyte/DBT/Metabase together

Re: Launch HN: Airbyte (YC W20) – Open-Source ELT (Fivetran/Stitch Alternative)

#89

Post by seektable »

crunchtoinfo wrote: post #86
yclurker wrote: post #32

The Open Source Fivetran alternative. Yay, it was about time! A simple license : MIT. Clear differentiation between free & paid plans. I am liking what I am seeing so far. One of our client is in advertising industry and is syncing data from 20 different API vendors to postgres. So I am one of your potential customer. However, there is a big problem I'm noticing with "Open source alternatives" lately on HN. I had to…

I can confirm that self-hosted is the best of both worlds: fully private and you control the access through your own infrastructure and all works in the browser. This simplicity is important for many small-medium companies. That is why my go-to tools for adhoc data slicing by non-technical users is https://www.metabase.com . I hope that Airbyte/dbt will be the ELT complement. And I hope to find a way to contribute.

Shameless plug: for adhoc reporting by non-technical users our https://www.seektable.com also can be considered (esp. if your users heavily use pivot tables), it is closed-source but has a fully functional free accounts & self-hosted version.

Re: Launch HN: Airbyte (YC W20) – Open-Source ELT (Fivetran/Stitch Alternative)

#90

Post by bloodjuggles »

Awesome tooling, great to see some open source work in this space.

How do you handle Testing/Data Reconciliation with Airbyte?

How do I know if I have successfully transferred 100/100 records for the day? Is there a pattern that you can recommend here or a batch row count id that is recorded that can be queried against the source for confirmation that the correct amount of rows were added for x batch?

Post reply on HN