Meltano: ELT for the DataOps era
meltano.com
Meltano: ELT for the DataOps era
1–10 of 34 posts
Re: Meltano: ELT for the DataOps era
#2They have the right idea, it's a real need and a major opportunity. Airbyte[1] is a few steps ahead. I also expect that the Airflow community will also wade into these waters at some point.
Re: Meltano: ELT for the DataOps era
#3They have the right idea, it's a real need and a major opportunity. Airbyte[1] is a few steps ahead. I also expect that the Airflow community will also wade into these waters at some point. 1. https://airbyte.io/
Another one on the market is pipelinewise.
Meltano and pipelinewise are both ways how to orchestrate Singer.io taps and targets while Airbyte is its own thing.
EDIT: I also don't know how just released Airbyte can be ahead of something that's surely in production for a while.
Re: Meltano: ELT for the DataOps era
#4They have the right idea, it's a real need and a major opportunity. Airbyte[1] is a few steps ahead. I also expect that the Airflow community will also wade into these waters at some point. 1. https://airbyte.io/
Another one on the market is pipelinewise. Meltano and pipelinewise are both ways how to orchestrate Singer.io taps and targets while Airbyte is its own thing. EDIT: I also don't know how just released Airbyte can be ahead of something that's surely in production for a while.
I'm currently using Stitch (https://www.stitchdata.com/) to move data from a few sources to Redshift.
Since Stitch, Meltano and Pipelinewise all use Singer.io taps and targets under the hood, I wonder if there's any reason to choose one over the other?
Re: Meltano: ELT for the DataOps era
#5Earlier quoted context omitted.
Another one on the market is pipelinewise. Meltano and pipelinewise are both ways how to orchestrate Singer.io taps and targets while Airbyte is its own thing. EDIT: I also don't know how just released Airbyte can be ahead of something that's surely in production for a while.
I'm currently using Stitch ( https://www.stitchdata.com/ ) to move data from a few sources to Redshift. Since Stitch, Meltano and Pipelinewise all use Singer.io taps and targets under the hood, I wonder if there's any reason to choose one over the other?
Stich is SaaS only. That by nature makes it suitable for some uses, and unsuitable for others (like when you have a provision that your data can't leave your network or when you are in a company where adding a new vendor isn't a quick process.
Meltano and Pipelinewise are open source projects that someone built for themselves but are sharing. You can just start playing with it and change the code or whatever, but there's no support to pay for.
For example where I'm standing the best one would be "Stitch I can self-host for free for a PoC and then eventually engage vendor about a support contract while still self-hosting it for security reasons" but there doesn't seem to be anything like it.
Re: Meltano: ELT for the DataOps era
#6Looks like it will integrate with Airflow and dbt. I like to see this kind of synergy in the open-source community.
Re: Meltano: ELT for the DataOps era
#7They have the right idea, it's a real need and a major opportunity. Airbyte[1] is a few steps ahead. I also expect that the Airflow community will also wade into these waters at some point. 1. https://airbyte.io/
Another one on the market is pipelinewise. Meltano and pipelinewise are both ways how to orchestrate Singer.io taps and targets while Airbyte is its own thing. EDIT: I also don't know how just released Airbyte can be ahead of something that's surely in production for a while.
Singer.io seems to run the data plane on json -- is there an Apache Arrow equiv for this ecosystem/problem, or something like it?
Re: Meltano: ELT for the DataOps era
#8There's a good intro to this project from last summer or so on the Data Engineering Podcast. It's actually a project within Gitlab iirc
Re: Meltano: ELT for the DataOps era
#9There's a good intro to this project from last summer or so on the Data Engineering Podcast. It's actually a project within Gitlab iirc
Yep - here's the link to the podcast [0]. And you're correct, it is a project with GitLab. For the past year or so there's been 1 person on the project, but there are now 2 (today is when I switch from the GitLab Data team to Meltano) and a 3rd will be joining us in a week!
[0] https://www.dataengineeringpodcast.com/meltano-data-integrat...
Re: Meltano: ELT for the DataOps era
#10Looks like it will integrate with Airflow and dbt. I like to see this kind of synergy in the open-source community.
You're correct, it currently does integrate with Airflow and dbt. I'm a big fan of both having used them basically every day for the past 3 years. The dbt community has done wonders for me personally and professionally and we're hoping to help build upon the dbt community (and others) to include open source data integration.