Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Meltano: ELT for the DataOps era

meltano.com

Post reply on HN
Print view View original

11–20 of 34 posts

Re: Meltano: ELT for the DataOps era

#11

Post by tayloramurphy »

rahimnathwani wrote: post #4
glogla wrote: post #3

Earlier quoted context omitted.

Another one on the market is pipelinewise. Meltano and pipelinewise are both ways how to orchestrate Singer.io taps and targets while Airbyte is its own thing. EDIT: I also don't know how just released Airbyte can be ahead of something that's surely in production for a while.

I'm currently using Stitch ( https://www.stitchdata.com/ ) to move data from a few sources to Redshift. Since Stitch, Meltano and Pipelinewise all use Singer.io taps and targets under the hood, I wonder if there's any reason to choose one over the other?

It's dependent on your use case, but the three examples you've listed here all have a slightly different approach to the market. Stitch (as mentioned in a comment below) is SaaS only. Pipelinewise is open source but as far as I know have no plans to build a company around it. With Meltano, we're aiming to grow the project and community and eventually build a business around it in a similar manner to what GitLab has done. Our docs[0] have more information about our current focus and roadmap if you're curious.

[0] https://meltano.com/docs/#focus

Re: Meltano: ELT for the DataOps era

#12

Post by tayloramurphy »

lmeyerov wrote: post #7
glogla wrote: post #3

Earlier quoted context omitted.

Another one on the market is pipelinewise. Meltano and pipelinewise are both ways how to orchestrate Singer.io taps and targets while Airbyte is its own thing. EDIT: I also don't know how just released Airbyte can be ahead of something that's surely in production for a while.

Singer.io seems to run the data plane on json -- is there an Apache Arrow equiv for this ecosystem/problem, or something like it?

I'm not aware of any. I did just open this issue[0] in the Meltano project to open discussion with the team/community. It could be an interesting iteration on the Singer Spec[1] if we find that users are interested in it and it helps solve some bottleneck challenges.

[0] https://gitlab.com/meltano/meltano/-/issues/2616 [1] https://github.com/singer-io/getting-started/blob/master/doc...

Re: Meltano: ELT for the DataOps era

#13

Post by shapiro92 »

before I try yet another ETL tool. How does this work with datasets that do not come from 3rd party providers like salesforce etc? I have had to build ETL pipelines for highly customized datasets either row level based or xml with I would say tricky code as the nesting or flows were not so simple and a lot of data missing.

How would Meltano or the other mention tools handle this?

Example is EDIFACT or FHIR or BDT

Re: Meltano: ELT for the DataOps era

#14

Post by estsauver »

shapiro92 wrote: post #13

before I try yet another ETL tool. How does this work with datasets that do not come from 3rd party providers like salesforce etc? I have had to build ETL pipelines for highly customized datasets either row level based or xml with I would say tricky code as the nesting or flows were not so simple and a lot of data missing. How would Meltano or the other mention tools handle this? Example is EDIFACT or FHIR or BDT

I can speak for singer taps which this is based on for the EL bits. Our postgres etl works quite well. Logical replication broke down quite quickly, but primary keys + updated_at keys are working very well for us. I can't speak about XML though.

Re: Meltano: ELT for the DataOps era

#15

Post by pchal »

tayloramurphy wrote: post #10
brylie wrote: post #6

Looks like it will integrate with Airflow and dbt. I like to see this kind of synergy in the open-source community.

You're correct, it currently does integrate with Airflow and dbt. I'm a big fan of both having used them basically every day for the past 3 years. The dbt community has done wonders for me personally and professionally and we're hoping to help build upon the dbt community (and others) to include open source data integration.

Any plans to integrate with dagster for pipeline orchestration (instead of airflow)?

Re: Meltano: ELT for the DataOps era

#16

Post by tayloramurphy »

pchal wrote: post #15
tayloramurphy wrote: post #10

Earlier quoted context omitted.

You're correct, it currently does integrate with Airflow and dbt. I'm a big fan of both having used them basically every day for the past 3 years. The dbt community has done wonders for me personally and professionally and we're hoping to help build upon the dbt community (and others) to include open source data integration.

Any plans to integrate with dagster for pipeline orchestration (instead of airflow)?

Yep! We have this issue[0] open. I'm also interested in getting Prefect and GitLab CI as orchestrators too.

[0] https://gitlab.com/meltano/meltano/-/issues/2393

Re: Meltano: ELT for the DataOps era

#17

Post by lmeyerov »

tayloramurphy wrote: post #12
lmeyerov wrote: post #7

Earlier quoted context omitted.

Singer.io seems to run the data plane on json -- is there an Apache Arrow equiv for this ecosystem/problem, or something like it?

I'm not aware of any. I did just open this issue[0] in the Meltano project to open discussion with the team/community. It could be an interesting iteration on the Singer Spec[1] if we find that users are interested in it and it helps solve some bottleneck challenges. [0] https://gitlab.com/meltano/meltano/-/issues/2616 [1] https://github.com/singer-io/getting-started/blob/master/doc...

yeah we do not push an etl pipeline through json unless we have to (and generally cannot), most etl-scale data engineering we do is almost all arrow/parquet/orc/protobuf etc, and slow legacy, odbc/json, which is streams that we turn into typed and compact data. I think json fine for command/metadata layers though, esp early on, but pretty core to what I look for an etl/streaming tool is out-of-the-box foundations for the data plane

the good news is the implicitly typed json examples look arrow friendly, so users can to/from_json if they don't care about data speed/quality like when prototyping and not think about it. there may be other data-engineering-friendly formats that'd work too.

prefect, dask, and friends solve it by abstracting over it. you can send whatever you want.. and it happens to be friendly to dataframes (pydata) / compact & typed data. but there projects seem to be more about source/sink, so encouraging structure by default would be helpful...

Re: Meltano: ELT for the DataOps era

#18

Post by dataminded »

glogla wrote: post #3
dataminded wrote: post #2

They have the right idea, it's a real need and a major opportunity. Airbyte[1] is a few steps ahead. I also expect that the Airflow community will also wade into these waters at some point. 1. https://airbyte.io/

Another one on the market is pipelinewise. Meltano and pipelinewise are both ways how to orchestrate Singer.io taps and targets while Airbyte is its own thing. EDIT: I also don't know how just released Airbyte can be ahead of something that's surely in production for a while.

Easy -- Meltano has not been in production. They are both relatively new tools.

Re: Meltano: ELT for the DataOps era

#19

Post by edgarrmondragon »

lmeyerov wrote: post #17
tayloramurphy wrote: post #12

Earlier quoted context omitted.

I'm not aware of any. I did just open this issue[0] in the Meltano project to open discussion with the team/community. It could be an interesting iteration on the Singer Spec[1] if we find that users are interested in it and it helps solve some bottleneck challenges. [0] https://gitlab.com/meltano/meltano/-/issues/2616 [1] https://github.com/singer-io/getting-started/blob/master/doc...

yeah we do not push an etl pipeline through json unless we have to (and generally cannot), most etl-scale data engineering we do is almost all arrow/parquet/orc/protobuf etc, and slow legacy, odbc/json, which is streams that we turn into typed and compact data. I think json fine for command/metadata layers though, esp early on, but pretty core to what I look for an etl/streaming tool is out-of-the-box foundations for…

AFAIK Meltano uses JSON only in the interface between a tap (source) and a target, to communicate schema, state and records.

It's up to the target what it does with the JSON messages it receives, so you can for example have a target-avro that takes JSON records and outputs them as an Avro file and translates the JSON schema to the corresponding Avro schema.

Re: Meltano: ELT for the DataOps era

#20

Post by smilliken »

estsauver wrote: post #14
shapiro92 wrote: post #13

before I try yet another ETL tool. How does this work with datasets that do not come from 3rd party providers like salesforce etc? I have had to build ETL pipelines for highly customized datasets either row level based or xml with I would say tricky code as the nesting or flows were not so simple and a lot of data missing. How would Meltano or the other mention tools handle this? Example is EDIFACT or FHIR or BDT

I can speak for singer taps which this is based on for the EL bits. Our postgres etl works quite well. Logical replication broke down quite quickly, but primary keys + updated_at keys are working very well for us. I can't speak about XML though.

Why did logical replication break down for you?
Post reply on HN