Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Meltano: ELT for the DataOps era

meltano.com

Post reply on HN
Print view View original

21–30 of 34 posts

Re: Meltano: ELT for the DataOps era

#21

Post by tayloramurphy »

shapiro92 wrote: post #13

before I try yet another ETL tool. How does this work with datasets that do not come from 3rd party providers like salesforce etc? I have had to build ETL pipelines for highly customized datasets either row level based or xml with I would say tricky code as the nesting or flows were not so simple and a lot of data missing. How would Meltano or the other mention tools handle this? Example is EDIFACT or FHIR or BDT

We're working on an SDK[0] for building taps that should make it much easier to build to the Singer spec with all of the features out of the box. In theory if you can write some python against whatever you're pulling data out of, then it can work within the Singer ecosystem and Meltano. It's nearly ready to go but we'd love feedback if you decide to test it out!

[0] https://gitlab.com/meltano/singer-sdk

Re: Meltano: ELT for the DataOps era

#24

Post by tayloramurphy »

kbd wrote: post #22

This has been called "ETL" for as long as I can remember. Could someone explain why they're using "ELT"?

This article[0] from Census is a good primer on the difference and what's driving it. But the TL;DR is that storage is cheap and it's easier to change SQL-based transformations using something like dbt[1] than to re-extract the data when the business logic changes.

[0] https://blog.getcensus.com/dbt-the-etl-elt-disrupter/ [1] https://www.getdbt.com/

Re: Meltano: ELT for the DataOps era

#25

Post by tayloramurphy »

dataminded wrote: post #18
glogla wrote: post #3

Earlier quoted context omitted.

Another one on the market is pipelinewise. Meltano and pipelinewise are both ways how to orchestrate Singer.io taps and targets while Airbyte is its own thing. EDIT: I also don't know how just released Airbyte can be ahead of something that's surely in production for a while.

Easy -- Meltano has not been in production. They are both relatively new tools.

The GitLab Data Team is running Meltano in production[0]. We're currently extracting Zoom data with it and have plans for several more extractors (Slack, GMail, PTO by Roots, EdCast, and a few more). I just made this MR[1] to update the list of Extractors to include Zoom too.

[0] https://about.gitlab.com/handbook/business-ops/data-team/pla... [1] https://gitlab.com/gitlab-com/www-gitlab-com/-/merge_request...

Re: Meltano: ELT for the DataOps era

#26

Post by JPKab »

kbd wrote: post #22

This has been called "ETL" for as long as I can remember. Could someone explain why they're using "ELT"?

While there are technical differences with transforming the data within it's storage location/database, I've interacted with my former employer's board of directors enough to know there's a bigger, rather obnoxious reason for this:

VC valuations in Silicon Valley for traditional "ETL" tools isn't great. The term "ETL" has baggage of Talend and other big enterprise tools that have an aura of "not being user friendly".

"ELT" is intended to signal that a tool is trying really hard to not be like those sad, difficult to use, dirty "ETL" tools.

That's the real reason. There really isn't an actual difference in how the tools work. Traditional ETL tools are perfectly capable of loading the data and initiating the transform in the database it's being stored in. This has been done for decades.

I have been repeatedly lectured by senior strategy team executives about using the proper vernacular to ensure perception of value is maximized in the trendy, buzzword driven VC world. So fucking stupid.

Re: Meltano: ELT for the DataOps era

#27

Post by DouweM »

tayloramurphy wrote: post #25
dataminded wrote: post #18

Earlier quoted context omitted.

Easy -- Meltano has not been in production. They are both relatively new tools.

The GitLab Data Team is running Meltano in production[0]. We're currently extracting Zoom data with it and have plans for several more extractors (Slack, GMail, PTO by Roots, EdCast, and a few more). I just made this MR[1] to update the list of Extractors to include Zoom too. [0] https://about.gitlab.com/handbook/business-ops/data-team/pla... [1] https://gitlab.com/gitlab-com/www-gitlab-com/-/merge_request...

And the GitLab Data Team is not alone! The Meltano Slack community (link on the homepage) is about 800 strong right now, and every day we've got people discussing their production deployments and helping new users set up their own.

PS. Like Taylor, I'm on the Meltano team at GitLab.

Re: Meltano: ELT for the DataOps era

#28

Post by nojito »

kbd wrote: post #22

This has been called "ETL" for as long as I can remember. Could someone explain why they're using "ELT"?

Marketing. There's been a huge marketing push of tools that designed to automate the SQL portion of data pipeline.

Doesn't make much sense to me, because anyone who's been in the data space quickly sees the rampant marketing of different tools that all do the same thing but simply change the order of the pipeline.

Re: Meltano: ELT for the DataOps era

#29

Post by kpierce »

kbd wrote: post #22

This has been called "ETL" for as long as I can remember. Could someone explain why they're using "ELT"?

Keeping a raw version of the data make it easier to change in the future. If you use s3 as a data lake you can transform later into the shape you want it. Also can use tools like presto where schema is on read instead of hardcoded so transform step is moved to another time.

Re: Meltano: ELT for the DataOps era

#30

Post by snidane »

kbd wrote: post #22

This has been called "ETL" for as long as I can remember. Could someone explain why they're using "ELT"?

ELT means: let's focus on moving data between from system A to system B and skip the transform part, which is a business of whoever has access to the target system. Which is usually a sql database so you can then do arbitrary data transforms there.

In other words just get the data in house into a queryable form and worry about the T part later.

Post reply on HN