Will Airbyte support MySQL as destination?
Yes! it is on our short term roadmap: https://github.com/airbytehq/airbyte/issues/1483
Also, we are OSS so if you want to contribute, we can guide you through it!
Launch HN: Airbyte (YC W20) – Open-Source ELT (Fivetran/Stitch Alternative)
31–40 of 90 posts
Re: Launch HN: Airbyte (YC W20) – Open-Source ELT (Fivetran/Stitch Alternative)
#32The Open Source Fivetran alternative. Yay, it was about time! A simple license : MIT. Clear differentiation between free & paid plans. I am liking what I am seeing so far. One of our client is in advertising industry and is syncing data from 20 different API vendors to postgres. So I am one of your potential customer.
However, there is a big problem I'm noticing with "Open source alternatives" lately on HN. I had to mention this.
Even a simple installation of airbyte on my local machine fails :( I tried docker-compose up!
I simply wanna know why a basic example is not working on an important day of your company ? :) Is this a genuine mistake ? Sorry, this feedback will sound harsh but companies are taking words 'open source' for a complete ride. It's a great marketing trick. Gets you plenty of eyeballs, good will & trust to begin with. Then later we figure it's not even self hostable.
Here is a bad example that you may not want to follow : Supabase "The Open Source Firebase Alternative". The product is not self hostable despite calling themselves open source firebase all over internet. The Founders of Supabase have been disingenuous not to address self hosting[1][2] and its a been long time since their launch. The self hosting section on their website[3] doesn't provide any details on how to self host and they are careless enough to even mention "how to migrate away" from Supabase in that section.
[1] : https://github.com/supabase/supabase/discussions/219#discuss... [2] : https://github.com/supabase/supabase/issues/85#issuecomment-... [3] : https://supabase.io/docs/guides/platform#self-hosting
Re: Launch HN: Airbyte (YC W20) – Open-Source ELT (Fivetran/Stitch Alternative)
#33How do you differentiate from Meltano?
The main difference is that Meltano is based on Singer while Airbyte is based on the Airbyte Protocol.
We don’t believe Singer is a good building block, because it requires a significant time investment from its users to compensate for the absence of centralized enforcement of the Singer protocol. Since it is not enforced, there is often no guarantee that any pair of Singer connectors are compatible. It defeats the point of a specification. All taps live in their own repo, and all contributions are made to address the contributor’s case, not the general use case. The lack of standard makes it very difficult to maintain all those connectors, and you end up with a majority of Singer taps being out of date.
Airbyte doesn’t have the same data protocol as Singer (but we are compatible). Our goal is to make building and maintaining new connectors a lot easier than it is with Singer, and therefore Meltano. That’s why we were able to ramp up our connectors (46 now) within just 5 months, while Meltano is focused on fixing the issues with Singer. We think it’s much harder to patch over Singer and reverse course on an abandonware project than it is to start from the ground up with these issues in mind. We wouldn’t be surprised if Meltano starts supporting Airbyte connectors in the future.
We detail these differences here: https://docs.airbyte.io/faq/differences-with.../meltano-vs-a...
Re: Launch HN: Airbyte (YC W20) – Open-Source ELT (Fivetran/Stitch Alternative)
#34Is there a guide to set it up behind a reverse proxy with https? I have it running with Nginx but it won't load, it insist in using /8001/api. How do I change the API endpoint?
Re: Launch HN: Airbyte (YC W20) – Open-Source ELT (Fivetran/Stitch Alternative)
#35The Open Source Fivetran alternative. Yay, it was about time! A simple license : MIT. Clear differentiation between free & paid plans. I am liking what I am seeing so far. One of our client is in advertising industry and is syncing data from 20 different API vendors to postgres. So I am one of your potential customer. However, there is a big problem I'm noticing with "Open source alternatives" lately on HN. I had to…
Sorry that it doesn't work :( Murphy law on launch day I would say...
Right now all our users self-host and the whole project is meant to be self-hosted for data privacy and security reasons.
Do you want to join our slack (https://slack.airbyte.io)? We can help you on the resolution!
Re: Launch HN: Airbyte (YC W20) – Open-Source ELT (Fivetran/Stitch Alternative)
#36The Open Source Fivetran alternative. Yay, it was about time! A simple license : MIT. Clear differentiation between free & paid plans. I am liking what I am seeing so far. One of our client is in advertising industry and is syncing data from 20 different API vendors to postgres. So I am one of your potential customer. However, there is a big problem I'm noticing with "Open source alternatives" lately on HN. I had to…
Sorry that it doesn't work :( Murphy law on launch day I would say... Right now all our users self-host and the whole project is meant to be self-hosted for data privacy and security reasons. Do you want to join our slack ( https://slack.airbyte.io )? We can help you on the resolution!
Traceback (most recent call last):
File "site-packages/urllib3/connectionpool.py", line 677, in urlopen
File "site-packages/urllib3/connectionpool.py", line 392, in _make_request
File "http/client.py", line 1252, in request
File "http/client.py", line 1298, in _send_request
File "http/client.py", line 1247, in endheaders
File "http/client.py", line 1026, in _send_output
File "http/client.py", line 966, in send
File "site-packages/docker/transport/unixconn.py", line 43, in connectRe: Launch HN: Airbyte (YC W20) – Open-Source ELT (Fivetran/Stitch Alternative)
#37Is there a guide to set it up behind a reverse proxy with https? I have it running with Nginx but it won't load, it insist in using /8001/api. How do I change the API endpoint?
You can. You can play the the env variable: API_URL
For example if you have a reverse proxy that serves both the webapp and the api you can just launch with: API_URL=/api/v1/ docker-compose up
Re: Launch HN: Airbyte (YC W20) – Open-Source ELT (Fivetran/Stitch Alternative)
#38I do a lot of ETL over the years and watched the video and read the intro. I apologize but I haven't a chance to read through the full documentation. My question is, a lot of data I pull are relational and hierarchical, how would I pass variables along to related connectors? And is there a way to wait for a parent connection to finish before the child connector runs? I've build many ETL over the years and it's not ea…
It is not possible as of now but we are about to start integrating with DAG managers (Airflow, Dagster, Prefect...). That will give the possibility to schedule the connectors with the proper dependencies. See: https://github.com/airbytehq/airbyte/issues/836
Correct me if I'm wrong, but Airbyte is mainly an EL to be used with other schedulers? The scheduler will have to provide, internally or externally, the logic and parameters to the nodes that drive Airbyte connectors?
Re: Launch HN: Airbyte (YC W20) – Open-Source ELT (Fivetran/Stitch Alternative)
#39Earlier quoted context omitted.
Sorry that it doesn't work :( Murphy law on launch day I would say... Right now all our users self-host and the whole project is meant to be self-hosted for data privacy and security reasons. Do you want to join our slack ( https://slack.airbyte.io )? We can help you on the resolution!
Traceback (most recent call last): File "site-packages/urllib3/connectionpool.py", line 677, in urlopen File "site-packages/urllib3/connectionpool.py", line 392, in _make_request File "http/client.py", line 1252, in request File "http/client.py", line 1298, in _send_request File "http/client.py", line 1247, in endheaders File "http/client.py", line 1026, in _send_output File "http/client.py", line 966, in send File "…
Thanks for posting! I don't have enough context to help you solve.
Do you want to send a screenshot of your terminal? michel [@] airbyte.io
Re: Launch HN: Airbyte (YC W20) – Open-Source ELT (Fivetran/Stitch Alternative)
#40Earlier quoted context omitted.
It is not possible as of now but we are about to start integrating with DAG managers (Airflow, Dagster, Prefect...). That will give the possibility to schedule the connectors with the proper dependencies. See: https://github.com/airbytehq/airbyte/issues/836
Correct me if I'm wrong, but Airbyte is mainly an EL to be used with other schedulers? The scheduler will have to provide, internally or externally, the logic and parameters to the nodes that drive Airbyte connectors?
we are still figuring out how we want to integrate with other schedulers.
One option would be that you configure your source/destination with the Airbyte UI or API and with the external scheduler you just reference to a connection object with an id.
We need to run some experimentations and talk to the community to see what makes the most sense. If you have some opinion / scenarios, do you want to write them in the ticket?