If you want to have a look at the open-source GitHub project, here's the link: https://github.com/airbytehq/airbyte
How to build a custom data connector in 2 hours
21–30 of 33 posts
Re: How to build a custom data connector in 2 hours
#22What is a connector?
It's a module which extracts data from a source (an API, database, etc..) or pushes data to a destination (warehouse, blob storage, API, etc..). They're ubiquitous in data engineering & analytics workflows where users often need to consolidate their data in one database or warehouse to build dashboards, run analyses, etc..
Re: How to build a custom data connector in 2 hours
#23Earlier quoted context omitted.
Data engineers, data scientist & analytics engineers. People that be users of Airbyte but also can be contributors. For data engineers, the objectives is to help them empower other roles to be data fluent and have the freedom to move data without bugging data engineers
How suited is Airbyte for multi-tennant use-cases? I might have a project coming up to build a semi-analytics solution where you would obviously want to allow the users to easily connect to their data sources, and Airbyte looks like it might be a good fit for that (for now my rough idea was to build some plumbing around Singer taps). Are you aware of any users that use it for such purposes and are there any specific…
We do support separate workspaces but we don't have auth yet. However because there is an API available, you can always build the account segmentation and permissionning at your app level and only interact with Airbyte using the API.
Re: How to build a custom data connector in 2 hours
#24Was hoping for something about rapid prototyping of electrical connectors.
This is a fun story in that vein: http://www.bigmessowires.com/2016/06/04/db-19-resurrecting-a...
Re: How to build a custom data connector in 2 hours
#25Cool!
We built something similar for on-premise data sources, in Clojure. https://basil.net
Re: How to build a custom data connector in 2 hours
#26Was hoping for something about rapid prototyping of electrical connectors.
That's what I was expecting as well. Guessing that "connector" an industry term that I'm just not aware of. Seems like some pre-made adapter patterns for ETL's from browsing the docs.
Re: How to build a custom data connector in 2 hours
#27What is a connector?
It's a module which extracts data from a source (an API, database, etc..) or pushes data to a destination (warehouse, blob storage, API, etc..). They're ubiquitous in data engineering & analytics workflows where users often need to consolidate their data in one database or warehouse to build dashboards, run analyses, etc..
I already responded to a different comment in this thread, but you seem to be knowledgable on the subject. Is this a tool to assist with ETL's between otherwise incompatible sources/destinations?
Re: How to build a custom data connector in 2 hours
#28Was hoping for something about rapid prototyping of electrical connectors.
I actually would love a service that did this for something mortally/individually affordable.
Re: How to build a custom data connector in 2 hours
#29Re: How to build a custom data connector in 2 hours
#30It took a substantial fraction of two hours for the page to load.