Meltano: ELT for the DataOps era • Hacker News Board

Re: Meltano: ELT for the DataOps era

#31

Post by glogla » Mon, Mar 01, 2021, 10:36 PM UTC

tayloramurphy wrote: post #12

lmeyerov wrote: post #7

Earlier quoted context omitted.

Singer.io seems to run the data plane on json -- is there an Apache Arrow equiv for this ecosystem/problem, or something like it?

I'm not aware of any. I did just open this issue[0] in the Meltano project to open discussion with the team/community. It could be an interesting iteration on the Singer Spec[1] if we find that users are interested in it and it helps solve some bottleneck challenges. [0] https://gitlab.com/meltano/meltano/-/issues/2616 [1] https://github.com/singer-io/getting-started/blob/master/doc...

Yep, definitely.

Then the holy grail would be to have bunch of taps-targets running in parallel for a single pipeline, each working on a subset of streams.

Re: Meltano: ELT for the DataOps era

#32

Post by failboat » Mon, Mar 01, 2021, 10:39 PM UTC

read through the homepage and not sure what value add this has over airflow? i must be completely misinterpreting the software.

Re: Meltano: ELT for the DataOps era

#33

Post by estsauver » Tue, Mar 02, 2021, 7:38 AM UTC

smilliken wrote: post #20

estsauver wrote: post #14

Earlier quoted context omitted.

I can speak for singer taps which this is based on for the EL bits. Our postgres etl works quite well. Logical replication broke down quite quickly, but primary keys + updated_at keys are working very well for us. I can't speak about XML though.

Why did logical replication break down for you?

Great question—-I don’t really know! We were using stitch and couldn’t really get a good answer.

Re: Meltano: ELT for the DataOps era

#34

Post by shapiro92 » Tue, Mar 02, 2021, 11:23 AM UTC

tayloramurphy wrote: post #21

shapiro92 wrote: post #13

before I try yet another ETL tool. How does this work with datasets that do not come from 3rd party providers like salesforce etc? I have had to build ETL pipelines for highly customized datasets either row level based or xml with I would say tricky code as the nesting or flows were not so simple and a lot of data missing. How would Meltano or the other mention tools handle this? Example is EDIFACT or FHIR or BDT

We're working on an SDK[0] for building taps that should make it much easier to build to the Singer spec with all of the features out of the box. In theory if you can write some python against whatever you're pulling data out of, then it can work within the Singer ecosystem and Meltano. It's nearly ready to go but we'd love feedback if you decide to test it out! [0] https://gitlab.com/meltano/singer-sdk

okay fair enough. but I guess then it doesn't offer anything extra for us as we have already the ETL Platform self made and we add the process with just an extra file. thanks!