Live data from Hacker News

Launch HN: Airbyte (YC W20) – Open-Source ELT (Fivetran/Stitch Alternative)

news.ycombinator.com

71–80 of 90 posts

Re: Launch HN: Airbyte (YC W20) – Open-Source ELT (Fivetran/Stitch Alternative)

#71

It looks very promising and is something I wish to scale Quantale( https://quantale.io/ ) to, It's a project I am working on, which has very similar logic of adding data sources and make it available for data visualization. Congrats on your launch!

Thanks!

Re: Launch HN: Airbyte (YC W20) – Open-Source ELT (Fivetran/Stitch Alternative)

#72
post #68

Earlier quoted context omitted.

Traceback (most recent call last): File "site-packages/urllib3/connectionpool.py", line 677, in urlopen File "site-packages/urllib3/connectionpool.py", line 392, in _make_request File "http/client.py", line 1252, in request File "http/client.py", line 1298, in _send_request File "http/client.py", line 1247, in endheaders File "http/client.py", line 1026, in _send_output File "http/client.py", line 966, in send File "…

Isn’t there a GitHub issue submission page?

Sure, we do: https://github.com/airbytehq/airbyte/issues/new/choose

Re: Launch HN: Airbyte (YC W20) – Open-Source ELT (Fivetran/Stitch Alternative)

#73
Very nice! Data wants to be free. Please check out this rosettable thing we open-sourced last year, it could come in handy: it allows real time ETL-like sync between mysql and postgres. https://github.com/francoisp/rosettable I've implemented the struts of a framework built around rosettable, with mautic davical as a proof of concept, it keeps a user defined map (one to one, many to one, one to many and many to many) in sync, live, both ways, surviving restarts even if there's been divergent edits on both sides while offline.

I havent released this as OS yet, but I'll put a demo online shortly.

For datalakes, I've also implemented something that could be useful: check out duplexRsync: https://github.com/francoisp/duplexRsync. I'll try to reach out via email to make sure you guys see this.

cheers! best, F

Re: Launch HN: Airbyte (YC W20) – Open-Source ELT (Fivetran/Stitch Alternative)

#75
post #53

I had a read through your docs but was unable to find any info on how you handle the sync of hard deletions from sources. We use Stitch at the moment and have found this to be a surprisingly hard problem to solve without binary log replication, and without full refreshes. For the moment we've ruled out binlogs as we use Aurora MySQL which requires you to binlog from the master , and tying my data warehouse replicatio…

We don’t today, we will tomorrow when we start working on CDC. You’re raising a good point with MySQL, we will need to take this limitation into account. Hopefully there is a workaround.

check this out, https://github.com/francoisp/rosettable. Uses binlog a reader to call postgres triggers, upon which you can build 2 way sync realtime.

Re: Launch HN: Airbyte (YC W20) – Open-Source ELT (Fivetran/Stitch Alternative)

#76

I had a read through your docs but was unable to find any info on how you handle the sync of hard deletions from sources. We use Stitch at the moment and have found this to be a surprisingly hard problem to solve without binary log replication, and without full refreshes. For the moment we've ruled out binlogs as we use Aurora MySQL which requires you to binlog from the master , and tying my data warehouse replicatio…

hello Matt, you might be interested in this: uses binlog to provide 2 way sync to postgres including hard delete detection, using postgres triggers. https://github.com/francoisp/rosettable

Re: Launch HN: Airbyte (YC W20) – Open-Source ELT (Fivetran/Stitch Alternative)

#77
post #50

Earlier quoted context omitted.

Your pricing page does not have any prices so it's hard to tell. I'll definitely keep an eye on your progress though. Your connector as container approach is interesting although i think IT departments may have issues with an on prem solution that spawns other containers by itself. One more thing - how will you protect yourself from let's say AWS forking you and selling a managed airbyte version?

The pricing depends on the number of connectors you need (but not the volume of data), the features you use, and the SLA. We already saw that there was a great willingness to pay, so we’re not really worried about that part. We will learn a great deal about that next year. Our sole focus in 2021 is the open-source part of the project, and becoming the new standard to replicate data. Regarding cloud providers as AWS,…

> The pricing depends on the number of connectors you need

It's not uncommon for even small companies to have as many as a dozen sources. This sort of pricing schema can hurt smaller companies with low data volume but many sources (think e-commerce, especially in the era of "headless").

Re: Launch HN: Airbyte (YC W20) – Open-Source ELT (Fivetran/Stitch Alternative)

#78

The Open Source Fivetran alternative. Yay, it was about time! A simple license : MIT. Clear differentiation between free & paid plans. I am liking what I am seeing so far. One of our client is in advertising industry and is syncing data from 20 different API vendors to postgres. So I am one of your potential customer. However, there is a big problem I'm noticing with "Open source alternatives" lately on HN. I had to…

hey yclurker, Supabase founder here. I'm sorry we haven't delivered a better self-hosting experience. This is clearly something we could do better. As I mentioned in your link[1], we're targeting a release of our CLI in Q1 (last week of march). > its a been long time since their launch It has been 8 months since our alpha launch*, and just over 1 month since our beta. I hope that sets some context, because personally…

@kiwicopple : I'm really sorry to say this, I understand you try your best to sound reasonable every time. It's just that you alone believe that we can't see past your replies. But it is pretty evident that you wont make Supabase self hostable. Dude, even proprietary closed products provide docker / docker-compose by default in their repos. Quite honestly, shame on Supabase for setting such a shady example.

Re: Launch HN: Airbyte (YC W20) – Open-Source ELT (Fivetran/Stitch Alternative)

#79
post #50

Earlier quoted context omitted.

The pricing depends on the number of connectors you need (but not the volume of data), the features you use, and the SLA. We already saw that there was a great willingness to pay, so we’re not really worried about that part. We will learn a great deal about that next year. Our sole focus in 2021 is the open-source part of the project, and becoming the new standard to replicate data. Regarding cloud providers as AWS,…

> The pricing depends on the number of connectors you need It's not uncommon for even small companies to have as many as a dozen sources. This sort of pricing schema can hurt smaller companies with low data volume but many sources (think e-commerce, especially in the era of "headless").

Very true. We're thinking about offering some kind of volume-based for small companies but with a limit that we won't go over. There are some solutions and we will try to offer the pricing the most adapted to the needs of our users. It will be an iterative process for sure!
Post reply on HN