51–60 of 90 posts
Also, while I see the E&L, what sort of T solutions do you offer or have planned on your roadmap?
We use Stitch at the moment and have found this to be a surprisingly hard problem to solve without binary log replication, and without full refreshes. For the moment we've ruled out binlogs as we use Aurora MySQL which requires you to binlog from the master, and tying my data warehouse replication to the master node concerns me.
In incremental mode, for common DBs e.g. MySQL / Postgres, will Airbyte pick up hard deletions ever?
I had a read through your docs but was unable to find any info on how you handle the sync of hard deletions from sources. We use Stitch at the moment and have found this to be a surprisingly hard problem to solve without binary log replication, and without full refreshes. For the moment we've ruled out binlogs as we use Aurora MySQL which requires you to binlog from the master , and tying my data warehouse replicatio…
You’re raising a good point with MySQL, we will need to take this limitation into account. Hopefully there is a workaround.
Awesome project. When compared to major vendor solutions such as Azure Data Factory, is your core advantage the breadth of connectors and open-source core? Are you concerned that other such providers offer polish like no-code web applications to broaden product reach beyond programmers? Also, while I see the E&L, what sort of T solutions do you offer or have planned on your roadmap?
The breadth of connectors is a large part of it, as well as the fact that we are by design multi-cloud, so we limit the vendor lock-in.
We are not too worried about other providers using our technology. The goal is to make Airbyte the open-source standard for EL, and every time a company builds on top of Airbyte we become more of a standard. The data market is so big and keeps growing, so there is room for more than one player. Who knows -- maybe one day Azure will use “Powered by Airbyte” to offer more connectors to their customers.
For the T part, it will depend on your destination. If you’re using a warehouse, DBT is an amazing tool, so we will deepen our integration with them. If you’re using other processing technologies, then we will see where the community brings us.
Earlier quoted context omitted.
Your pricing page does not have any prices so it's hard to tell. I'll definitely keep an eye on your progress though. Your connector as container approach is interesting although i think IT departments may have issues with an on prem solution that spawns other containers by itself. One more thing - how will you protect yourself from let's say AWS forking you and selling a managed airbyte version?
This is a key question. Licensing via GPL-3 would provide some protection against this, so can you explain why you went with MIT?
With this in mind, GPL would bring no additional benefit, it might actually be a friction point for adoption in some organizations.
I work for a big financial institution and we are a actively evaluating tools in this space. Do you include connectors for Salesforce?
The fundamental challenge of open-source ETL is that high-quality connectors require understanding and working around all kinds of corner cases in the API of each data source. It’s very hard to get open source contributors to do this kind of work; it’s a real slog. Hence at Fivetran we’ve always stuck with the commercial route.
A customer perspective from a mid-stage CFO: I like saving money, but prefer to pay for software solutions like this directly. I pay you, and you make sure this set of connectors {in and out} continues to durably work. Meanwhile, our engineers can focus on building our product.
for me, work like ELT( https://fivetran.com or https://getcensus.com) are the type of work that no engineer in the world will get a promotion from.
data|software|back|platform|etc Engineer's time is better spend on something else than that.
Earlier quoted context omitted.
A customer perspective from a mid-stage CFO: I like saving money, but prefer to pay for software solutions like this directly. I pay you, and you make sure this set of connectors {in and out} continues to durably work. Meanwhile, our engineers can focus on building our product.
Couldn't agree more. for me, work like ELT( https://fivetran.com or https://getcensus.com ) are the type of work that no engineer in the world will get a promotion from. data|software|back|platform|etc Engineer's time is better spend on something else than that.
Every engineers we talked to want it out of their plate. Which is why we believe it should be commoditized with an open-source standard.
Earlier quoted context omitted.
You can. You can play the the env variable: API_URL For example if you have a reverse proxy that serves both the webapp and the api you can just launch with: API_URL=/api/v1/ docker-compose up
I just effortlessly synced Freshdesk to Postgres in 20 minutes. Thank you.